You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/02/03 13:30:52 UTC
[GitHub] [flink] azagrebin opened a new pull request #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
azagrebin opened a new pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999
## What is the purpose of the change
In release 1.10, with FLIP-49, we introduced significant changes to the TaskExecutor memory model and it's related configuration options / logics.
It is very important that we clearly state the changes and potential effects, and guide our users to tune their clusters with the new configuration for both new setups and migrations of previous setups.
## Brief change log
- Add memory model description and configuration guide
- Add tuning guide for various features
- Add troubleshooting guide
- Add migration guide from pre-FLIP-49 configuration
## Verifying this change
```
cd flink/docs/docker
./run.sh
./build_docs.sh -p
```
open `localhost:4000`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (docs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764310
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
Review comment:
```suggestion
<td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily. See also <a href="#managed-memory">how to migrate managed memory</a>.</td>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376653721
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
Review comment:
```suggestion
In addition, Flink dependencies, such as Hadoop, may consume more direct or native memory in certain setups.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376652016
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
Review comment:
```suggestion
| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink operators and user code |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374450324
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
Review comment:
Not necessarily in flink-conf.yaml. It can also be '-D' parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738050
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
Review comment:
Capitalization of heading is not consistent with the headings of the other pages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738116
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
Review comment:
```suggestion
It is recommended to configure the [*total process memory*](mem_setup.html#configure-total-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374141409
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
Review comment:
```suggestion
*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762617
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to [increase the JVM metaspace`](../config.html#taskmanager-memory-jvm-metaspace-size).
Review comment:
```suggestion
You can try to increase the [JVM metaspace](../config.html#taskmanager-memory-jvm-metaspace-size).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738024
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
Review comment:
```suggestion
title: "Memory Tuning Guide"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376522355
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+<span class="label label-info">Note</span> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
Review comment:
```suggestion
to specify [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory) explicitly.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738201
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
Review comment:
```suggestion
or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set the [*managed memory*](mem_setup.html#managed-memory) to zero.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374447643
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
Review comment:
I think it would be better to align the description with the figure above, to avoid potential confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738151
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
Review comment:
```suggestion
<span class="label label-info">Note</span> If you configure the *total Flink memory*, then Flink will implicitly add the JVM memory components
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738299
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
+how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,
+Flink will gracefully spill to disk.
Review comment:
```suggestion
then Flink will gracefully spill to disk.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656415
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :------------------------------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) | 64Mb |
+{:.table-bordered}
+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
Review comment:
```suggestion
All of the components listed above can still be configured explicitly for local execution.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376655953
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
Review comment:
```suggestion
Flink explicitly adds the following memory related JVM arguments while starting the task executor process.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376651611
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
Review comment:
```suggestion
| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink operators and user code |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374570324
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose the following options are set:
+- total Flink memory = 1000mb,
+- task heap = 100mb,
+- network min = 64mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
Review comment:
I did not want to list all the options (there is also managed memory btw). This forces to do more calculation comparing to just reading the next sentences
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376765197
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are `taskmanager.heap.size` or `taskmanager.heap.mb`.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: `mesos.resourcemanager.tasks.mem` which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (`taskmanager.memory.size`) was renamed to
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (`taskmanager.memory.fraction`)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
+This should prevent the killing of containers on [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+### Other changes
+
+Additionally, the following changes have been made:
+* The [managed memory](mem_setup.html#managed-memory) is always off-heap now. The configuration option `taskmanager.memory.off-heap` is removed and will have no effect anymore.
+* The [managed memory](mem_setup.html#managed-memory) now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup.html#managed-memory) is always lazily allocated now. The configuration option `taskmanager.memory.preallocate` is removed and will have no effect anymore.
+
+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
+This is no longer available and the related configuration options (`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
+will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
+described further, to address these concerns.
+
+In streaming jobs which use [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend), the RocksDB
+native memory consumption should be accounted for as a part of the [managed memory](mem_setup.html#managed-memory) now.
+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:
Review comment:
```suggestion
The other direct or native off-heap memory consumers can now be addressed by the following new configuration options:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374475034
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
Review comment:
```suggestion
The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738266
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
Review comment:
```suggestion
This means that the [*managed memory*](../memory/mem_setup.html#managed-memory) configuration affects
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374471963
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for the containerized deployments
Review comment:
```suggestion
([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)) for the containerized deployments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374475810
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
Review comment:
```suggestion
| taskmanager.memory.preallocate | Pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147945023 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
* e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147945023) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658698
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
Review comment:
I don't fully understand this part. Please rephrase if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374143368
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
Review comment:
I would stick to a consistent capitalization. Sometimes you write "memory" and then again "Memory" in the headings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374143906
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
Review comment:
```suggestion
The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376765017
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are `taskmanager.heap.size` or `taskmanager.heap.mb`.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: `mesos.resourcemanager.tasks.mem` which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (`taskmanager.memory.size`) was renamed to
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (`taskmanager.memory.fraction`)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
Review comment:
Maybe we could add that one could turn this behaviour off if one wants the old behaviour.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374550841
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
Review comment:
True, I will just remove `in *flink-conf.yaml*`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374142356
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
Review comment:
```suggestion
*size* will override *fraction*, if both are set.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374146134
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
Review comment:
```suggestion
Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764141
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
Review comment:
```suggestion
It also references other chapters for more details about migrating to the new configuration options.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374471624
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) or its [components](mem_setup.html#detailed-memory-model)
Review comment:
```suggestion
([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
* e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-584288562
Thanks for the review @tillrohrmann
I addressed the comments
merged into master by 01a9ff99608e40ad893002eeaa1c87eb9eaa27c3 - 0cb8834aa5ab8cdbef30b3ab5a6572356f76b64a
merged into 1.10 by ff7e25cad70a74da1795c69fd651d788d5972f1c - 1212f662cd201af44f4b6507f789488d2fceb9af
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374132064
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
Review comment:
Same for "Job Manager"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376518007
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
Review comment:
```suggestion
Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of a job manager.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sjwiesman commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581542701
@azagrebin I have a few small comments on the placement of the TOC and other minutiae. I'm relying on Till and others to validate this for correctness but overall I think this looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374144323
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
Review comment:
```suggestion
<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376522259
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+<span class="label label-info">Note</span> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
Review comment:
```suggestion
As mentioned before in [total memory description](#configure-total-memory), another way to configure Flink's memory is
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374576450
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
Review comment:
I will add here as we say they are still valid for JM but not later where we say they are not available anymore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374464483
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
Review comment:
We have both "mb" and "Mb" in the examples. I think it would be better to consistently use one of them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374454620
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
Review comment:
```suggestion
* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374478542
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
Review comment:
Do we want to also add links for these two options?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762371
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
Review comment:
```suggestion
by increasing [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657005
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :------------------------------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) | 64Mb |
+{:.table-bordered}
+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
+*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory)
+has a default value of 128Mb only for the local execution mode.
+
+<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case.
+It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started
+local process is not controlled by Flink and depends on how you start the process.
+If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. *-Xmx*, *-Xms*.
Review comment:
```suggestion
If you want to control the JVM heap size, then you have to explicitly pass the corresponding arguments *-Xmx* and *-Xms* to the JVM.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374135681
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
Review comment:
```suggestion
<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376447038
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (‘taskmanager.memory.size’) was renamed to
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (‘taskmanager.memory.fraction’)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
+This should prevent the killing of containers on [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+### Other changes
+
+Additionally, the following changes have been made:
+* The [managed memory](mem_setup.html#managed-memory) is always off-heap now. The configuration option ‘taskmanager.memory.off-heap’ is removed and will have no effect anymore.
+* The [managed memory](mem_setup.html#managed-memory) now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup.html#managed-memory) is always lazily allocated now. The configuration option ‘taskmanager.memory.preallocate’ is removed and will have no effect anymore.
+
+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
+This is no longer available and the related configuration options (`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
+will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
+described further, to address these concerns.
+
+In streaming jobs which use [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend), the RocksDB
+native memory consumption should be accounted for as a part of the [managed memory](mem_setup.html#managed-memory) now.
+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:
+* Task off-heap memory ([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size))
+* Framework off-heap memory ([`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size))
+* JVM metaspace ([`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size))
+* JVM overhead (see also [detailed new memory model](mem_setup.html#detailed-memory-model))
+
+<span class="label label-info">Note</span> The job manager still has container cut-off memory configuration options.
+The mentioned configuration options remain valid for the job manager in the same way as before.
+
+## Default Configuration in flink-conf.yaml
+
+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+in the default ‘flink-conf.yaml’. The value is also increased from 1024Mb to 1568Mb.
Review comment:
and here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374446425
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
Review comment:
These two links do not work for me. Why do we have `-1` in the urls?
Is it a mistake or something wrong with my building?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376654799
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
Review comment:
```suggestion
The minimum and maximum values have defaults and can be explicitly configured via their corresponding configuration options.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656644
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :------------------------------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) | 64Mb |
+{:.table-bordered}
+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
+*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory)
Review comment:
```suggestion
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [*managed memory*](mem_setup.html#managed-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764233
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
Review comment:
```suggestion
This chapter shortly lists all changes to Flink's memory configuration options introduced with the 1.10 release.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762362
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
Review comment:
```suggestion
The exception usually indicates that the JVM heap is too small. You can try to increase the JVM heap size
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376517708
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
Review comment:
Nit: In the first sentence we use italics for the release version and in the next sentence we don't use italics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376655217
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
Review comment:
```suggestion
Notice if you configure the same minimum and maximum value then the component's size is fixed to this value.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376518724
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
Review comment:
```suggestion
Otherwise, the simplest way to setup memory in Flink is to configure either of the following two options:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376655446
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
Review comment:
```suggestion
then the network memory will be 128Mb because the size derived from fraction is 100Mb which is less than the minimum.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738276
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
Review comment:
```suggestion
the performance of your applications. Flink will use as much [managed memory](../memory/mem_setup.html#managed-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762705
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to [increase the JVM metaspace`](../config.html#taskmanager-memory-jvm-metaspace-size).
+
+## IOException: Insufficient number of network buffers
+
+The exception usually indicates that the size of the configured [network memory](mem_setup.html#detailed-memory-model)
+is not big enough. You can try to increase the *network memory* by adjusting the following options:
+* [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min)
+* [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max)
+* [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction)
+
+## Container Memory exceeded
+
+If a task executor container tries to allocate memory beyond its requested size (Yarn, Mesos or Kubernetes),
+this usually indicates that Flink has not reserved enough native memory. You can observe that either from an external
+monitoring system or from the error messages when a container gets killed by deployment environment.
Review comment:
```suggestion
monitoring system or from the error messages when a container gets killed by the deployment environment.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738251
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
Review comment:
```suggestion
Some operations work directly on raw data.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376763721
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
Review comment:
```suggestion
were removed or their semantics changed. This guide will help you to migrate the memory configuration from Flink <= 1.9 to >= 1.10.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762497
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
Review comment:
```suggestion
The exception usually indicates that the JVM *direct memory* limit is too small or that there is a *direct memory leak*.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738187
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
Review comment:
```suggestion
will dictate the optimal memory configurations for your cluster.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738121
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
Review comment:
```suggestion
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376649591
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+<span class="label label-info">Note</span> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+<span class="label label-info">Note</span> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction).
+
+*Size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<span class="label label-info">Note</span> You can also adjust the [framework off-heap memory](mem_detail.html#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](mem_detail.html#jvm-parameters).
+
+<span class="label label-info">Note</span> Although, native non-direct memory usage can be accounted for as a part of the
Review comment:
```suggestion
<span class="label label-info">Note</span> Although, native non-direct memory usage can be accounted for as part of the
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738064
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
Review comment:
For standalone deployments, it is recommended to configure [*total Flink memory*](....). *Total Flink memory* defines with how much memory Flink itself will be started.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657117
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
Review comment:
Not consistent capitalization wrt the other titles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376651897
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
Review comment:
```suggestion
| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, used for sorting, hash tables, caching of intermediate results and RocksDB state backend |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581415151
I will duplicate files for the later Chinese translation before merging.
cc @xintongsong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657725
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
Review comment:
Capitalization of heading is not consistent with the headings of the other pages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374493406
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<strong>Note:</strong> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryException`'s because Flink knows precisely
Review comment:
The class name is `OutOfMemoryError`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374474881
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
Review comment:
```suggestion
* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657520
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
Review comment:
```suggestion
In addition to the [main memory setup guide](mem_setup.html), this section explains how to configure the task executor's memory for different workloads and deployments.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374452675
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
Review comment:
Do we really want this link?
Currently, it links to the DataSet API page, which seems implying that it's only used in DataSet API, while in fact it works the same way for blink batch sql jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738059
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
Review comment:
```suggestion
Additionally, you can adjust the *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657182
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
Review comment:
```suggestion
title: "Memory Tuning Guide"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374450062
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
Review comment:
Same problem of '-1' in urls for these two lines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374454712
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
Review comment:
```suggestion
*Size* will override *fraction*, if both are set.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374150059
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
+see also [JVM Parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, the native non-direct usage of memory can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the actual *direct memory* allocation will have
+a bigger JVM limit in this case.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
Review comment:
I would suggest to make this a separate page. This will shorten the page a bit and make it easier to digest for the reader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374138060
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
Review comment:
```suggestion
* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size-1))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r375976477
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (‘taskmanager.memory.size’) was renamed to
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (‘taskmanager.memory.fraction’)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
+This should prevent the killing of containers on [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+### Other changes
+
+Additionally, the following changes have been made:
+* The [managed memory](mem_setup.html#managed-memory) is always off-heap now. The configuration option ‘taskmanager.memory.off-heap’ is removed and will have no effect anymore.
+* The [managed memory](mem_setup.html#managed-memory) now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup.html#managed-memory) is always lazily allocated now. The configuration option ‘taskmanager.memory.preallocate’ is removed and will have no effect anymore.
+
+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
+This is no longer available and the related configuration options (`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
+will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
+described further, to address these concerns.
+
+In streaming jobs which use [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend), the RocksDB
+native memory consumption should be accounted for as a part of the [managed memory](mem_setup.html#managed-memory) now.
+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:
+* Task off-heap memory ([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size))
+* Framework off-heap memory ([`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size))
+* JVM metaspace ([`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size))
+* JVM overhead (see also [detailed new memory model](mem_setup.html#detailed-memory-model))
+
+<span class="label label-info">Note</span> The job manager still has container cut-off memory configuration options.
+The mentioned configuration options remain valid for the job manager in the same way as before.
+
+## Default Configuration in flink-conf.yaml
+
+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
Review comment:
Why no backticks here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738229
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
Review comment:
```suggestion
RocksDB is set up to limit native memory allocation to the size of [*managed memory*](mem_setup.html#managed-memory).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376518384
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
Review comment:
should direct memory be italic similar to the other components?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376654386
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
Review comment:
```suggestion
The size of these components always has to be between its minimum and maximum value. Otherwise Flink's startup will fail.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374463771
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
Review comment:
```suggestion
then the network memory will be 128Mb because the size derived from fraction is 100mb and it is less than the minimum.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376653654
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
Review comment:
```suggestion
An increased memory need can be caused by a particular deployment environment or job structure, like high parallelism.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657554
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374476080
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
Review comment:
```suggestion
| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374485798
##########
File path: docs/ops/state/state_backends.zh.md
##########
@@ -115,6 +120,8 @@ RocksDBStateBackend 的适用场景：
然而，这也意味着使用 RocksDBStateBackend 将会使应用程序的最大吞吐量降低。
所有的读写都必须序列化、反序列化操作，这个比基于堆内存的 state backend 的效率要低很多。
+Check also recommendations about the [task executor memory configuration](../memory/mem_tuning.html#rocksdb-state-backend) for the RocksDBStateBackend.
Review comment:
Translation:
请同时参考 \[Task Executor 内存配置\](../memory/mem_tuning.html#rocksdb-state-backend) 中关于 RocksDBStateBackend 的建议。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376654087
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
Review comment:
```suggestion
This section describes the configuration details of the following options which can be a fraction of the
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376448937
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
@sjwiesman what's your opinion? is it typo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738155
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
Review comment:
```suggestion
in order to derive the *total process memory* for the container requests.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374553862
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
Review comment:
I think the reference can be still useful but true, we could also mention Blink batch jobs. I did not find what would make sense to refer to in docs for Blink. Alternatively, we can mention Blink w/o reference:
`Batch jobs (Blink planner or [DataSet API](../../dev/batch)) ...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374556962
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
Review comment:
Would this be easier to understand?
`Although, native non-direct memory usage can be accounted for as a part of the
*framework off-heap memory* or *task off-heap memory*, it will result in a higher JVM's *direct memory* limit in this case.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374137993
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
Review comment:
```suggestion
* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size-1))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374162024
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
Review comment:
It is not highlighted as a link in the usual way then
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374144563
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
Review comment:
```suggestion
and only recommended to be changed if you are sure that the Flink framework needs more memory.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374132499
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
Review comment:
```suggestion
Check [Job Manager related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376520301
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
Review comment:
```suggestion
For containerized deployments this value corresponds to the size of requested containers. See also
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764000
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
Review comment:
```suggestion
<span class="label label-info">Note</span> Before version 1.10, Flink did not require that memory related options are set at all
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374473559
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,70 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<strong>Note:</strong> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to increase the [JVM metaspace](../config.html#taskmanager-memory-jvm-metaspace-size).
Review comment:
```suggestion
You can try to [increase the JVM metaspace](../config.html#taskmanager-memory-jvm-metaspace-size).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374138618
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
Review comment:
I would use back-ticks for configuration options in the whole documentation. I won't comment this anymore from here on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581415786
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 (Mon Feb 03 13:34:16 UTC 2020)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374568012
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
Review comment:
`is` was related to `the size` but we can change it to `the sizes` then if it is more clear
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374456543
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
Review comment:
This sentence is a bit hard to understand for me.
Not sure if there is indeed a problem, or it's just my limited English skill.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581566520
Thanks for the review @sjwiesman ! I have addressed the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738234
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
Review comment:
```suggestion
Therefore, it is important to reserve enough *managed memory* to accommodate the state of your application. If you disable the RocksDB memory control,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374463537
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
Review comment:
To be more precise, we probably want to mention that none of other memory components are configured.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374669258
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
Review comment:
offline discussion outcome: align with the rest of doc and use back-ticks
we can think about more readable config option links as a follow-up in the whole docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762445
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
Review comment:
```suggestion
is advanced and should only be changed if you are sure that the Flink framework itself needs more memory.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764014
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
Review comment:
```suggestion
as they all had default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374142091
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
Review comment:
```suggestion
* or computed as a fraction of *total Flink memory* via [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762474
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to [increase the JVM metaspace`](../config.html#taskmanager-memory-jvm-metaspace-size).
+
+## IOException: Insufficient number of network buffers
+
+The exception usually indicates that the size of the configured [network memory](mem_setup.html#detailed-memory-model)
+is not big enough. You can try to increase the *network memory* by adjusting the following options:
+* [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min)
+* [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max)
+* [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction)
+
+## Container Memory exceeded
Review comment:
Inconsistent capitalization of the headings. I suggest to follow a consistent style.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376659087
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
Review comment:
```suggestion
It declares how much memory should be assigned in total to the Flink *JVM process* and corresponds to the size of the requested container.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376652814
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
Review comment:
```suggestion
| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for JVM overhead, e.g. threads, code cache, garbage collection space, etc. It is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738006
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
Review comment:
```suggestion
If you see an `IllegalConfigurationException` thrown from `TaskExecutorProcessUtils`, it usually indicates
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738175
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
Review comment:
```suggestion
then it can happen that the deployment environment kills the offending container.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376518548
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
Review comment:
```suggestion
options are relevant. See [local execution](mem_detail.html#local-execution) for more details.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r375974361
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
Plural is without `'` just `s`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374548790
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
Review comment:
The options, which have `-1` at the end of their links, were duplicated in the main configuration page. They were listed as common options and then as TM or TM mem specific. After docs restructure, they got deduplicated. I will remove `-1` for them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581415151
I will duplicate files for the later Chinese translation before merging and reference old model (#11004).
cc @xintongsong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762342
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
Review comment:
```suggestion
that there is either an invalid configuration value (e.g. negative memory size, fraction that is greater than 1, etc.)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sjwiesman commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374248530
##########
File path: docs/ops/memory/index.md
##########
@@ -0,0 +1,24 @@
+---
+nav-title: 'Memory configuration'
Review comment:
```suggestion
nav-title: 'Memory Configuration'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on issue #10999: [FLINK-15143] Docs for
FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581909826
Thanks for the review @tillrohrmann
I addressed the comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374465989
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose the following options are set:
+- total Flink memory = 1000mb,
+- task heap = 100mb,
+- network min = 64mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
Review comment:
Would it be better also give absolute values to managed memory, task / framework off-heap memory in the example, so we can clearly show what size the network memory will be derived as.
E.g.,
- total Flink memory = 1000mb,
- task heap = 200mb,
- task off-heap = 200mb,
- framework heap = 200mb
- framework off-heap = 200mb
- network min = 64mb,
- network max = 256mb,
- network fraction = 0.1
Then the network memory size is 200mb.
I think the current example is correct, but may not be that easy to understand for users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764501
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are `taskmanager.heap.size` or `taskmanager.heap.mb`.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: `mesos.resourcemanager.tasks.mem` which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
Review comment:
Inconsistent spelling of containerized in this document.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656155
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
Review comment:
```suggestion
If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374484879
##########
File path: docs/ops/state/state_backends.zh.md
##########
@@ -91,6 +93,9 @@ FsStateBackend 适用场景:
- 状态比较大、窗口比较长、key/value 状态比较大的 Job。
- 所有高可用的场景。
+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
Review comment:
Translation:
建议同时将 \[managed memory\](../memory/mem_setup.html#managed-memory) 设为0，以保证将最大限度的内存分配给 JVM 上的用户代码。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374444629
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
Review comment:
It seems the linked page does not have the anchor `#jobmanager`. This link jumps to the beginning of the configuration pate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376446932
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (‘taskmanager.memory.size’) was renamed to
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (‘taskmanager.memory.fraction’)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
+This should prevent the killing of containers on [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+### Other changes
+
+Additionally, the following changes have been made:
+* The [managed memory](mem_setup.html#managed-memory) is always off-heap now. The configuration option ‘taskmanager.memory.off-heap’ is removed and will have no effect anymore.
+* The [managed memory](mem_setup.html#managed-memory) now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup.html#managed-memory) is always lazily allocated now. The configuration option ‘taskmanager.memory.preallocate’ is removed and will have no effect anymore.
+
+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
+This is no longer available and the related configuration options (`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
+will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
+described further, to address these concerns.
+
+In streaming jobs which use [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend), the RocksDB
+native memory consumption should be accounted for as a part of the [managed memory](mem_setup.html#managed-memory) now.
+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:
+* Task off-heap memory ([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size))
+* Framework off-heap memory ([`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size))
+* JVM metaspace ([`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size))
+* JVM overhead (see also [detailed new memory model](mem_setup.html#detailed-memory-model))
+
+<span class="label label-info">Note</span> The job manager still has container cut-off memory configuration options.
+The mentioned configuration options remain valid for the job manager in the same way as before.
+
+## Default Configuration in flink-conf.yaml
+
+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
Review comment:
Indeed, I overlooked old options w/o links
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764290
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
Review comment:
```suggestion
<td>On-heap <i>managed memory</i> is no longer supported. See also <a href="#managed-memory">how to migrate managed memory</a>.</td>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658698
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
Review comment:
I don't fully understand this part. Please rephrase if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376521856
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
Review comment:
```suggestion
This means that one of the following option subsets has to be configured explicitly:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762569
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
Review comment:
```suggestion
the [JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376650063
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
Review comment:
We could call this page also simply "Memory Model"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376519757
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
Review comment:
```suggestion
is given to Flink itself. The *total Flink memory* splits up into *JVM heap*, [*managed memory size*](#managed-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376659440
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
Review comment:
```suggestion
See [detailed Memory Model](mem_setup.html#detailed-memory-model) for more information.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658323
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
Review comment:
For standalone deployments, it is recommended to configure [*total Flink memory*](....). *Total Flink memory* defines with how much memory Flink itself will be started.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658323
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
Review comment:
For standalone deployments, it is recommended to configure [*total Flink memory*](....). *Total Flink memory* defines with how much memory Flink itself will be started.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376650970
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
Review comment:
I would rephrase the sentence below the figure slightly: "Notice that the user code's native non-direct memory usage can also be accounted for as part of the task off-heap memory."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738074
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
Review comment:
```suggestion
The *total Process memory* is not relevant because the *JVM overhead* is not controlled by Flink or its deployment environment,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738125
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
Review comment:
```suggestion
It declares how much memory should be assigned in total to the Flink *JVM process* and corresponds to the size of the requested container.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374471075
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose the following options are set:
+- total Flink memory = 1000mb,
+- task heap = 100mb,
+- network min = 64mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :------------------------ | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+
+See also [detailed memory model](#detailed-memory-model).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :----------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br /> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) | 64Mb |
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](#task-operator-heap-memory) and
+*task off-heap memory* is considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](#managed-memory)
Review comment:
```suggestion
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](#managed-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374455972
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
Review comment:
```suggestion
<strong>Note:</strong> You can also customize the [framework off-heap memory](#framework-memory). This option is advanced
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374447296
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
Review comment:
```suggestion
is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory](#managed-memory), and direct memory.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374483636
##########
File path: docs/ops/state/state_backends.zh.md
##########
@@ -71,6 +71,8 @@ MemoryStateBackend 适用场景：
- 本地开发和调试。
- 状态很小的 Job，例如：由每次只处理一条记录的函数（Map、FlatMap、Filter 等）构成的 Job。Kafka Consumer 仅仅需要非常小的状态。
+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
Review comment:
Translation:
建议同时将 \[managed memory\](../memory/mem_setup.html#managed-memory) 设为0，以保证将最大限度的内存分配给 JVM 上的用户代码。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374475775
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
Review comment:
```suggestion
| taskmanager.memory.off-heap | On-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sjwiesman commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374253494
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<strong>Note:</strong> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+the RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+[Managed memory](../memory/mem_setup.html#managed-memory) helps Flink to run the batch operators efficiently.
+Therefore, the size of the [managed memory](mem_setup.html#managed-memory) can affect the performance of [batch jobs](../../dev/batch).
+If it makes sense, Flink will try to allocate and use as much *managed memory* as configured for batch jobs but not beyond its limit.
+It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations.
+If Flink runs out of [managed memory](../memory/mem_setup.html#managed-memory), it utilizes disk space.
+Using [managed memory](../memory/mem_setup.html#managed-memory), some operations can be performed directly
+on the raw data without having to deserialize the data to convert it into Java objects. All in all,
+[managed memory](../memory/mem_setup.html#managed-memory) improves the robustness and speed of the system.
Review comment:
> Flink's [batch](../../dev/batch) operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently. In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. This means that managed memory configurations have practical effects on the performance of your applications. Flink will attempt to allocate and use as much managed memory as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryException`'s because Flink knows precisely how much memory it has to leverage. If the managed memory is not sufficient, Flink will gracefully spill to disk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738218
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
Review comment:
```suggestion
This ensures that the maximum amount of memory will be allocated for user code on the JVM.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374148529
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
+see also [JVM Parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, the native non-direct usage of memory can be accounted for as a part of the
Review comment:
```suggestion
<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147945023 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:bac32011ba81cc76a62308ae0334ce5731b11cce Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:bac32011ba81cc76a62308ae0334ce5731b11cce
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
* e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147945023) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966)
* bac32011ba81cc76a62308ae0334ce5731b11cce UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374144929
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
+see also [JVM Parameters](#jvm-parameters).
Review comment:
```suggestion
see also [JVM parameters](#jvm-parameters).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657892
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
Review comment:
```suggestion
Additionally, you can adjust the *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376651259
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
Review comment:
```suggestion
The following table lists all of Flink's memory components and references the configuration options
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374473870
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,70 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<strong>Note:</strong> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to increase the [JVM metaspace](../config.html#taskmanager-memory-jvm-metaspace-size).
Review comment:
This paragraph have two "JVM metaspace"s linking to different pages.
As a user, I would thought these two links are the same thing, and will not click on the second.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581415151
I will duplicate files for the later Chinese translation before merging and reference old model from #11004.
cc @xintongsong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738283
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
```suggestion
as configured for the execution of batch jobs. This prevents `OutOfMemoryErrors` because Flink knows precisely
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738289
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
+how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,
Review comment:
```suggestion
how much memory it can use. If the [*managed memory*](../memory/mem_setup.html#managed-memory) is not large enough,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147945023 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:bac32011ba81cc76a62308ae0334ce5731b11cce Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147377727 TriggerType:PUSH TriggerID:bac32011ba81cc76a62308ae0334ce5731b11cce
Hash:bac32011ba81cc76a62308ae0334ce5731b11cce Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4832 TriggerType:PUSH TriggerID:bac32011ba81cc76a62308ae0334ce5731b11cce
Hash:0cb8834aa5ab8cdbef30b3ab5a6572356f76b64a Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:0cb8834aa5ab8cdbef30b3ab5a6572356f76b64a
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
* e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147945023) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966)
* bac32011ba81cc76a62308ae0334ce5731b11cce Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147377727) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4832)
* 0cb8834aa5ab8cdbef30b3ab5a6572356f76b64a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374148304
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
+see also [JVM Parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, the native non-direct usage of memory can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the actual *direct memory* allocation will have
Review comment:
```suggestion
*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374141685
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
Review comment:
```suggestion
The size of *managed memory* can be
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374555150
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
Review comment:
I will change to `adjust`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374131503
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
Review comment:
```suggestion
The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738042
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374474897
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
Review comment:
```suggestion
* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764718
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are `taskmanager.heap.size` or `taskmanager.heap.mb`.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: `mesos.resourcemanager.tasks.mem` which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
Review comment:
```suggestion
The JVM heap memory is also used by the heap state backends ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657182
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
Review comment:
```suggestion
title: "Memory Tuning Guide"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374460167
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
Review comment:
Broken link for jvm-overhead-max.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762696
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to [increase the JVM metaspace`](../config.html#taskmanager-memory-jvm-metaspace-size).
+
+## IOException: Insufficient number of network buffers
+
+The exception usually indicates that the size of the configured [network memory](mem_setup.html#detailed-memory-model)
+is not big enough. You can try to increase the *network memory* by adjusting the following options:
+* [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min)
+* [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max)
+* [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction)
+
+## Container Memory exceeded
+
+If a task executor container tries to allocate memory beyond its requested size (Yarn, Mesos or Kubernetes),
+this usually indicates that Flink has not reserved enough native memory. You can observe that either from an external
Review comment:
```suggestion
this usually indicates that Flink has not reserved enough native memory. You can observe this either by using an external
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374458913
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
Review comment:
I would suggest also add this notice inside the figure file.
Usually, when we have a star mark in the figure, we should include the corresponding footnote in the figure as well. This guarantees that the figure and the footnote always stay together, in case the figure is cited / reused in some other places. I've seen many times that people cite figures from Flink's official docs in their tech-blogs or presentation slides.
Minor: "notice" -> "Notice"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656507
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :------------------------------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) | 64Mb |
+{:.table-bordered}
+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
Review comment:
```suggestion
If they are not configured they are set to their default values. [*Task heap memory*](mem_setup.html#task-operator-heap-memory) and
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374462876
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
Review comment:
I would move this notice to after the first example.
Currently, when I read "For example" right after this notice, I was expecting an example of min=max.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376653164
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
Review comment:
I would remove these two sentences.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658755
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
Review comment:
```suggestion
It is recommended to configure the [*total process memory*](mem_setup.html#configure-total-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658619
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
Review comment:
```suggestion
The *total Process memory* is not relevant because the *JVM overhead* is not controlled by Flink or its deployment environment,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376763444
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
Review comment:
```suggestion
title: "Set up Task Executor Memory"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376763826
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
Review comment:
```suggestion
result in different sizes of memory components. If you try to reuse your Flink configuration from older versions
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376659280
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
Review comment:
```suggestion
<span class="label label-info">Note</span> If you configure the *total Flink memory*, then Flink will implicitly add the JVM memory components
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r375974361
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
Plural is without `'` just `s`
Maybe it's easier to just write `OutOfMemoryErrors`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sjwiesman commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374254363
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<strong>Note:</strong> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+the RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
Review comment:
```suggestion
RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374478983
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
+have no effect for task manager processes anymore but they still have the same semantics for the job manager process.
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) for standalone deployments
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for containerized deployments (Yarn or Mesos)
Review comment:
```suggestion
* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376443824
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
I think we can just remove plural
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376655579
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
Review comment:
```suggestion
It can also happen that the fraction is ignored if the sizes of the total memory and other components are defined.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374474077
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,70 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<strong>Note:</strong> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.
+
+## OutOfMemoryError: Direct buffer memory
+
+The exception usually indicates that the JVM *direct memory* limit is too small if there is no *direct memory leak*.
+You can try to increase this limit by adjusting [direct off-heap memory](mem_setup.html#detailed-memory-model).
+See also [how to configure off-heap memory](mem_setup.html#configure-off-heap-memory-direct-or-native) and
+[JVM arguments](mem_setup.html#jvm-parameters) which Flink sets.
+
+## OutOfMemoryError: Metaspace
+
+The exception usually indicates that [JVM metaspace](mem_setup.html#jvm-parameters) limit is configured too small.
+You can try to increase the [JVM metaspace](../config.html#taskmanager-memory-jvm-metaspace-size).
+
+## IOException: Insufficient number of network buffers
+
+The exception usually indicates that the size of the configured [network memory](mem_setup.html#detailed-memory-model)
+is not big enough.
+
Review comment:
Do we also want to say "try increasing network memory" and link to corresponding options here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376659440
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
Review comment:
```suggestion
See [detailed Memory Model](mem_setup.html#detailed-memory-model) for more information.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin merged pull request #10999: [FLINK-15143] Docs
for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin merged pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374480919
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
+have no effect for task manager processes anymore but they still have the same semantics for the job manager process.
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) for standalone deployments
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
+
+The previous option to configure managed memory size (‘taskmanager.memory.size’) was renamed to
+[taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) and deprecated.
+It is recommended to use the new option because the legacy one can be removed in future releases.
+
+### Fraction
+
+If not set explicitly, the managed memory could be previously specified as a fraction (‘taskmanager.memory.fraction’)
+of the total memory minus network memory and container cut-off (only for [Yarn](../deployment/yarn_setup.html) and
+[Mesos](../deployment/mesos.html) deployments). This option has been completely removed and will have no effect if still used.
+Please, use the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) instead.
+This new option will set the [managed memory](mem_setup.html#managed-memory) to the specified fraction of the
+[total Flink memory](mem_setup.html#configure-total-memory) if its size is not set explicitly by
+[taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size).
+
+### RocksDB state
+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.
+This should prevent the killing of containers on [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+### Other changes
+
+Additionally, the following changes have been made:
+* The [managed memory](mem_setup.html#managed-memory) is always off-heap now. The configuration option ‘taskmanager.memory.off-heap’ is removed and will have no effect anymore.
+* The [managed memory](mem_setup.html#managed-memory) now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup.html#managed-memory) is always lazily allocated now. The configuration option ‘taskmanager.memory.preallocate’ is removed and will have no effect anymore.
+
+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
+This is no longer available and the related configuration options (‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’)
+will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
+described further, to address these concerns.
+
+In streaming jobs which use [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend), the RocksDB
+native memory consumption should be accounted for as a part of the [managed memory](mem_setup.html#managed-memory) now.
+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:
+* Task off-heap memory ([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size))
+* Framework off-heap memory ([taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size))
+* JVM metaspace ([taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size))
+* JVM overhead (see also [detailed new memory model](mem_setup.html#detailed-memory-model))
+
+<strong>Note:</strong> The job manager still has container cut-off memory configuration options.
+The mentioned configuration options remain valid for the job manager in the same way as before.
+
+## Default Configuration in flink-conf.yaml
+
+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
Review comment:
```suggestion
The total memory (‘taskmanager.heap.size’) is replaced by [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sjwiesman commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374245612
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
Review comment:
Again, please move to line 30 below the first paragraph
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376649704
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+<span class="label label-info">Note</span> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+<span class="label label-info">Note</span> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction).
+
+*Size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<span class="label label-info">Note</span> You can also adjust the [framework off-heap memory](mem_detail.html#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](mem_detail.html#jvm-parameters).
+
+<span class="label label-info">Note</span> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it will result in a higher JVM's *direct memory* limit in this case.
Review comment:
```suggestion
*framework off-heap memory* or *task off-heap memory*, it would result in a higher JVM's *direct memory* limit.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658802
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
Review comment:
```suggestion
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376653351
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
Review comment:
```suggestion
Adjust them only if you are sure that Flink needs more memory for its internal data structures or operations.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376443824
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
Review comment:
I think we can just remove plural
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656856
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
+
+| **JVM Arguments** | **Value** |
+| :---------------------------------------- | :----------------------------------------- |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-Heap + Network Memory |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace |
+{:.table-bordered}
+<br/>
+
+See also [detailed memory model](#overview).
+
+## Local Execution
+If you start Flink locally on your machine as a single java program without creating a cluster (e.g. from your IDE)
+then all components are ignored except for the following:
+
+| **Memory component** | **Relevant options** | **Default value for the local execution** |
+| :------------------------------------------- | :-------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
+| Task heap | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | infinite |
+| Task off-heap | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | infinite |
+| Managed memory | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | 128Mb |
+| Network memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) | 64Mb |
+{:.table-bordered}
+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
+*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory)
+has a default value of 128Mb only for the local execution mode.
+
+<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case.
+It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started
Review comment:
```suggestion
The actual JVM heap size of the started
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374463869
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
Review comment:
```suggestion
It can also happen that the fraction is ignored if the size of the total memory and its other components are defined.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657892
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
Review comment:
```suggestion
Additionally, you can adjust the *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r375014551
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :----------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [taskmanager.memory.framework.heap.size](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](#task-operator-heap-memory) | [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](#managed-memory) | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) <br/> [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [taskmanager.memory.framework.off-heap.size](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option)) |
+| Task Off-heap Memory | [taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [taskmanager.memory.jvm-metaspace.size](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [taskmanager.memory.network.min](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [taskmanager.memory.network.max](../config.html#taskmanager-jvm-overhead-network-max) <br/> [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](#configure-total-memory) |
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+### Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<strong>Note:</strong> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+### Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#detailed-memory-model).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 64mb,
+- network max = 128mb,
+- network fraction = 0.1
+
+then the network memory will be 1000mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if the following options are:
+- total Flink memory = 1000mb,
+- network min = 128mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the fraction is 100mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the size of the total memory and its other components is defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose the following options are set:
+- total Flink memory = 1000mb,
+- task heap = 100mb,
+- network min = 64mb,
+- network max = 256mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
Review comment:
If we don't want to list all the options, we may also say that sum of configured values of the rest options is 600mb, leaving 1000 - 200 - 600 = 200mb for network memory.
I think the example should be concrete. Currently, we give the concrete value of total flink (1000mb) and task heap (100mb), but these values do not help showing how to derive a concrete network size due to absence of other sizes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374149674
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
+* or computed as a fraction of *total Flink memory* ([taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction))
+
+If both *size* and *fraction* are explicitly set, the *size* will be used and the *fraction* will be ignored.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure off-heap memory (direct or native)
+
+The off-heap memory, allocated in user code, should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase [framework off-heap memory](#framework-memory) but this option is advanced
+and recommended to be changed if you are sure that Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit controlled by JVM,
+see also [JVM Parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, the native non-direct usage of memory can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the actual *direct memory* allocation will have
+a bigger JVM limit in this case.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation.
Review comment:
```suggestion
to never exceed its configured size. Therefore, resizing network memory will not necessarily help in this situation.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374483006
##########
File path: docs/ops/state/state_backends.md
##########
@@ -74,6 +74,8 @@ The MemoryStateBackend is encouraged for:
- Local development and debugging
- Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer requires very little state.
+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
Review comment:
```suggestion
It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374578239
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
Review comment:
Indeed, I can try this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376762396
##########
File path: docs/ops/memory/mem_trouble.md
##########
@@ -0,0 +1,73 @@
+---
+title: "Troubleshooting"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
+or configuration conflicts. Check the documentation chapters related to the [memory components](mem_setup.html#detailed-memory-model)
+mentioned in the exception message.
+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
Review comment:
```suggestion
<span class="label label-info">Note</span> You can also increase the [framework heap memory](mem_setup.html#framework-memory) but this option
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738177
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
Review comment:
```suggestion
See also the description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374134896
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
Review comment:
I would remove everything after the second sentence.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764411
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
Review comment:
```suggestion
The container cut-off configuration options, [`containerized.heap-cutoff-ratio`](config.html#containerized-heap-cutoff-ratio)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374576450
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
Review comment:
I will add here as we say they are still valid for JM but not later where we say they are not available anymore for TM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581432351
<!--
Meta data
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147207707 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783 TriggerType:PUSH TriggerID:3ed9750b8441390bc9ca14babc2cc4bc302e4e31
Hash:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:83e8781a15350d710c03cc7fe2a9e4f00cbd5f51
Hash:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:f2ec29ad602092ba2fde6857677f8ce758bb2479 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147237088 TriggerType:PUSH TriggerID:f2ec29ad602092ba2fde6857677f8ce758bb2479
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147252899 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:94175b95319b633ed135a39d032d435b4c5eca77 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797 TriggerType:PUSH TriggerID:94175b95319b633ed135a39d032d435b4c5eca77
Hash:09865b1c436f1d03f82187b8a2f66a5abddfa6d5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:09865b1c436f1d03f82187b8a2f66a5abddfa6d5
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147262567 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:4ff824338b9bee9a678932f6c5bdad6e62582379 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799 TriggerType:PUSH TriggerID:4ff824338b9bee9a678932f6c5bdad6e62582379
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147226914 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795 TriggerType:PUSH TriggerID:1dafbfabae2a1d994437aa56ae0ae4687474c8d9
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147355414 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:bbc0d2c53b5eca13227e3251e0e354783422b51c Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818 TriggerType:PUSH TriggerID:bbc0d2c53b5eca13227e3251e0e354783422b51c
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147364447 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824 TriggerType:PUSH TriggerID:b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8
Hash:11780141de10ad68113eb133cb4740b4d4460642 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:11780141de10ad68113eb133cb4740b4d4460642
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147383775 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:16eeb758a4c5908166bf135e66f989611c268634 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834 TriggerType:PUSH TriggerID:16eeb758a4c5908166bf135e66f989611c268634
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147945023 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
Hash:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966 TriggerType:PUSH TriggerID:e34dafad44cbd3eb8cc62b0e0436f096a8f95db6
-->
## CI report:
* 3ed9750b8441390bc9ca14babc2cc4bc302e4e31 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147207707) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4783)
* 83e8781a15350d710c03cc7fe2a9e4f00cbd5f51 UNKNOWN
* a38bb3f58dd6c57b0943e3365d4a02c6fdbf5491 UNKNOWN
* f2ec29ad602092ba2fde6857677f8ce758bb2479 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147237088) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4796)
* 94175b95319b633ed135a39d032d435b4c5eca77 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147252899) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4797)
* 09865b1c436f1d03f82187b8a2f66a5abddfa6d5 UNKNOWN
* 4ff824338b9bee9a678932f6c5bdad6e62582379 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147262567) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4799)
* 1dafbfabae2a1d994437aa56ae0ae4687474c8d9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147226914) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4795)
* bbc0d2c53b5eca13227e3251e0e354783422b51c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147355414) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4818)
* b7df6003a9a529b0fc9382cf12465c3c8cb0c0b8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147364447) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4824)
* 11780141de10ad68113eb133cb4740b4d4460642 UNKNOWN
* 16eeb758a4c5908166bf135e66f989611c268634 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147383775) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4834)
* e34dafad44cbd3eb8cc62b0e0436f096a8f95db6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147945023) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4966)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376522496
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+<span class="label label-info">Note</span> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
Review comment:
```suggestion
It gives more control over the available JVM heap for Flink’s tasks and their [managed memory](#managed-memory).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374157825
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
Review comment:
Here I wanted to emphasise that it is for TM not JM. I can try to use <strong> closer to TM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764743
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Removed option</th>
+ <th class="text-left">Note</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.memory.fraction</h5></td>
+ <td>
+ Check the description of the new option <a href="../config.html#taskmanager-memory-managed-fraction)">taskmanager.memory.managed.fraction</a>.
+ The new option has different semantics and the value of the deprecated option usually has to be adjusted.
+ See also <a href="#managed-memory">how to migrate managed memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.off-heap</h5></td>
+ <td>On-heap <i>managed memory</i> is no longer supported, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.preallocate</h5></td>
+ <td>Pre-allocation is no longer supported and <i>managed memory</i> is always allocated lazily, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ </tbody>
+</table>
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Deprecated option</th>
+ <th class="text-left">Interpreted as</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td>
+ <ul>
+ <li><a href="../config.html#taskmanager-memory-flink-size">taskmanager.memory.flink.size</a> for <a href="../deployment/cluster_setup.html">standalone deployment</a></li>
+ <li><a href="../config.html#taskmanager-memory-process-size">taskmanager.memory.process.size</a> for containerised deployments</li>
+ </ul>
+ See also <a href="#total-memory-previously-heap-memory">how to migrate total memory</a>.
+ </td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.memory.size</h5></td>
+ <td><a href="../config.html#taskmanager-memory-managed-size">taskmanager.memory.managed.size</a>, see also <a href="#managed-memory">how to migrate managed memory</a>.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.min</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-min">taskmanager.memory.network.min</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.max</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-max">taskmanager.memory.network.max</a></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.network.memory.fraction</h5></td>
+ <td><a href="../config.html#taskmanager-memory-network-fraction">taskmanager.memory.network.fraction</a></td>
+ </tr>
+ </tbody>
+</table>
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
+and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
+but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are `taskmanager.heap.size` or `taskmanager.heap.mb`.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: `mesos.resourcemanager.tasks.mem` which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) for standalone deployments
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)
+
+It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.
+
+See also [how to configure total memory now](mem_setup.html#configure-total-memory).
+
+## JVM Heap Memory
+
+JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest
+which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory,
+see also [how to migrate managed memory](#managed-memory).
+
+Now, if only *total Flink memory* or *total process memory* is configured, then the JVM heap is also derived as the rest of
+what is left after subtracting all other components from the total memory, see also [how to configure total memory](mem_setup.html#configure-total-memory).
+
+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend) if it is chosen for streaming jobs.
+
+A part of the JVM heap is now always reserved for Flink framework
+([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
+See also [Framework memory](mem_setup.html#framework-memory).
+
+## Managed Memory
+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size
Review comment:
Capitalization
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374131962
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
Review comment:
Inconsistent casing. Sometimes Task Executor is written with capital letters and sometimes with lower case letters. I'd suggest to stick to one way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin edited a comment on issue #10999: [FLINK-15143]
Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on issue #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#issuecomment-581415151
I will duplicate files for the later Chinese translation before merging and reference old model (#11004).
cc @xintongsong @sjwiesman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374141883
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,252 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described further in a [separate section](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have explicitly configured task heap and managed memory, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may lead to deployment failures due to potential memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed memory
+
+This type of memory is native (off-heap) and explicitly managed by Flink. Use cases for *managed memory* are the following:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of managed memory can be
+* either configured explicitly ([taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size))
Review comment:
```suggestion
* either configured explicitly via [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374477623
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
Review comment:
These two tables look dazzling after compiled, at least in my browser.
I think it would be better if we draw the lines of the table units.
Shall we consider to use html tables instead of markdown tables?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658755
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
Review comment:
```suggestion
It is recommended to configure the [*total process memory*](mem_setup.html#configure-total-memory)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658802
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
Review comment:
```suggestion
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for containerized deployments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657725
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
Review comment:
Capitalization of heading is not consistent with the headings of the other pages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376657554
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738161
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
Review comment:
```suggestion
See [detailed Memory Model](mem_setup.html#detailed-memory-model) for more information.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376658619
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
Review comment:
```suggestion
The *total Process memory* is not relevant because the *JVM overhead* is not controlled by Flink or its deployment environment,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374547340
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
Review comment:
Indeed, the docs have been recently restructured. I will find a new link
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376656026
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
+| [JVM metaspace](#jvm-parameters) | [`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size) | Metaspace size of the Flink JVM process |
+| JVM Overhead | [`taskmanager.memory.jvm-overhead.min`](../config.html#taskmanager-memory-jvm-overhead-min) <br/> [`taskmanager.memory.jvm-overhead.max`](../config.html#taskmanager-memory-jvm-overhead-max) <br/> [`taskmanager.memory.jvm-overhead.fraction`](../config.html#taskmanager-memory-jvm-overhead-fraction) | Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a [capped fractionated component](#capped-fractionated-components) of the [total process memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+As you can see, the size of some memory components can be simply set by the respective option.
+Other components can be tuned using multiple options.
+
+## Framework Memory
+
+The *framework heap memory* and *framework off-heap memory* options are not supposed to be changed without a good reason.
+Adjust them only if you are sure that Flink needs more memory for some internal data structures or operations.
+It can be related to a particular deployment environment or job structure, like high parallelism.
+In addition, Flink dependencies, such as Hadoop may consume more direct or native memory in certain setups.
+
+<span class="label label-info">Note</span> Neither heap nor off-heap versions of framework and task memory are currently isolated within Flink.
+The separation of framework and task memory can be used in future releases for further optimizations.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
+
+* *Network memory* can be a fraction of the *total Flink memory*
+* *JVM overhead* can be a fraction of the *total process memory*
+
+See also [detailed memory model](#overview).
+
+The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
+The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 64Mb,
+- network max = 128Mb,
+- network fraction = 0.1
+
+then the network memory will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
+
+Notice if you configure the same maximum and minimum value it effectively means that its size is fixed to that value.
+
+If the component memory is not explicitly configured, then Flink will use the fraction to calculate the memory size
+based on the total memory. The calculated value is capped by its corresponding min/max options.
+For example, if only the following memory options are set:
+- total Flink memory = 1000Mb,
+- network min = 128Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+then the network memory will be 128Mb because the size derived from fraction is 100Mb and it is less than the minimum.
+
+It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
+In this case, the network memory is the rest of the total memory. The derived value still has to be within its min/max
+range otherwise the configuration fails. For example, suppose only the following memory options are set:
+- total Flink memory = 1000Mb,
+- task heap = 100Mb,
+- network min = 64Mb,
+- network max = 256Mb,
+- network fraction = 0.1
+
+All other components of the total Flink memory have default values, including the default managed memory fraction.
+Then the network memory is not the fraction (1000Mb x 0.1 = 100Mb) but the rest of the total Flink memory
+which will either be within the range 64-256Mb or fail.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while starting the task executor process,
+based on the configured or derived memory component sizes:
Review comment:
```suggestion
The values are based on the configured or derived memory component sizes:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376521335
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,133 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
+* Total process memory ([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](mem_detail.html#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory size](#managed-memory)
+and *direct memory*.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
Review comment:
I would put the config keys directly behind `task heap` and `managed memory` individually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738163
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
Review comment:
```suggestion
<strong>Warning:</strong> If Flink or the user code allocates off-heap (native) memory beyond its container size,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374454668
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
Review comment:
```suggestion
* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738182
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
Review comment:
```suggestion
When deploying a Flink streaming application, the configured [state backend](../state/state_backends.html) type
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376738237
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.
+
+* toc
+{:toc}
+
+## Configure memory for standalone deployment
+
+It is recommended to configure [total Flink memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)) or its [components](mem_setup.html#detailed-memory-model)
+for [standalone deployment](../deployment/cluster_setup.html) where you want to declare how much memory is given to Flink itself.
+Additionally, you can adjust *JVM metaspace* if it causes [problems](mem_trouble.html#outofmemoryerror-metaspace).
+
+The *total Process memory* is not relevant because *JVM overhead* is not controlled by Flink or deployment environment,
+only physical resources of the executing machine matter in this case.
+
+## Configure memory for containers
+
+It is recommended to configure [total process memory](mem_setup.html#configure-total-memory)
+([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)) for the containerized deployments
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+It declares how much memory in total should be assigned to the Flink *JVM process* and corresponds to the size of the requested container.
+
+<span class="label label-info">Note</span> If you configure the *total Flink memory* Flink will implicitly add JVM memory components
+to derive the *total process memory* and request a container with the memory of that derived size,
+see also [detailed Memory Model](mem_setup.html#detailed-memory-model).
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> If Flink or user code allocates unmanaged off-heap (native) memory beyond the container size
+ the job can fail because the deployment environment can kill the offending containers.
+</div>
+See also description of [container memory exceeded](mem_trouble.html#container-memory-exceeded) failure.
+
+## Configure memory for state backends
+
+When deploying a Flink streaming application, the type of [state backend](../state/state_backends.html) used
+will dictate the optimal memory configurations of your cluster.
+
+### Heap state backend
+
+When running a stateless job or using a heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+or [FsStateBackend](../state/state_backends.html#the-fsstatebackend), set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.
+
+### RocksDB state backend
+
+The [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) uses native memory. By default,
+RocksDB is setup to limit native memory allocation to the size of the [managed memory](mem_setup.html#managed-memory).
+Therefore, it is important to reserve enough *managed memory* for your state use case. If you disable the default RocksDB memory control,
+task executors can be killed in containerized deployments if RocksDB allocates memory above the limit of the requested container size
+(the [total process memory](mem_setup.html#configure-total-memory)).
+See also [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed).
+
+## Configure memory for batch jobs
+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
Review comment:
```suggestion
Flink's batch operators leverage [*managed memory*](../memory/mem_setup.html#managed-memory) to run more efficiently.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376652467
##########
File path: docs/ops/memory/mem_detail.md
##########
@@ -0,0 +1,149 @@
+---
+title: "Detailed Memory Model"
+nav-parent_id: ops_mem
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section gives a detailed description of all components in Flink’s memory model of task executor.
+Check [memory configuration guide](mem_setup.html) for the basic memory setup.
+
+* toc
+{:toc}
+
+## Overview
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+The following table lists all memory components, depicted above, and references Flink configuration options
+which affect the size of the respective components:
+
+| **Component** | **Configuration options** | **Description** |
+| :-------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Framework Heap Memory](#framework-memory) | [`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size) | JVM heap memory dedicated to Flink framework (advanced option) |
+| [Task Heap Memory](mem_setup.html#task-operator-heap-memory) | [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) | JVM heap memory dedicated to Flink application to run operators and user code |
+| [Managed memory](mem_setup.html#managed-memory) | [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) <br/> [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction) | Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend |
+| [Framework Off-heap Memory](#framework-memory) | [`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink framework (advanced option) |
+| [Task Off-heap Memory](mem_setup.html#configure-off-heap-memory-direct-or-native) | [`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size) | [Off-heap direct (or native) memory](mem_setup.html#configure-off-heap-memory-direct-or-native) dedicated to Flink application to run operators |
+| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
Review comment:
```suggestion
| Network Memory | [`taskmanager.memory.network.min`](../config.html#taskmanager-memory-network-min) <br/> [`taskmanager.memory.network.max`](../config.html#taskmanager-memory-network-max) <br/> [`taskmanager.memory.network.fraction`](../config.html#taskmanager-memory-network-fraction) | Direct memory used for data exchange between tasks. It is a [capped fractionated component](#capped-fractionated-components) of the [total Flink memory](mem_setup.html#configure-total-memory) |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374563823
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -0,0 +1,251 @@
+---
+title: "Setup Task Executor Memory"
+nav-parent_id: ops_mem
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink provides efficient workloads on top of the JVM by tightly controlling the memory usage of its various components.
+While the community strives to offer sensible defaults to all configurations, the full breadth of applications
+that users deploy on Flink means this isn't always possible. To provide the most production value to our users,
+Flink allows both high level and fine-grained tuning of memory allocation within clusters.
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.
+
+## Configure Total Memory
+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.
+
+If you configure *total process memory* you declare how much memory in total should be assigned to the Flink *JVM process*.
+For the containerized deployments it corresponds to the size of the requested container, see also
+[how to configure memory for containers](mem_tuning.html#configure-memory-for-containers)
+([Kubernetes](../deployment/kubernetes.html), [Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
+
+Another way to setup the memory is to set [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
+([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)).
+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+<strong>Note:</strong> Explicitly configuring both *total process memory* and *total Flink memory* is not recommended.
+It may lead to deployment failures due to potential memory configuration conflicts. Additional configuration
+of other memory components also requires caution as it can produce further configuration conflicts.
+The conflict can occur e.g. if the sum of sub-components does not add up to the total configured memory or size of some
+component is outside of its min/max range, see also [the detailed memory model](#detailed-memory-model).
+
+## Configure Heap and Managed Memory
+
+As mentioned before in [total memory description](#configure-total-memory), another way to setup memory in Flink is
+to specify explicitly both [task heap](#task-operator-heap-memory) and [managed memory](#managed-memory).
+It gives more control over the available JVM heap to Flink’s tasks and its [managed memory](#managed-memory).
+
+The rest of the memory components will be adjusted automatically, based on default values or additionally configured options.
+[Here](#detailed-memory-model) are more details about the other memory components.
+
+<strong>Note:</strong> If you have configured the task heap and managed memory explicitly, it is recommended to set neither
+*total process memory* nor *total Flink memory*. Otherwise, it may easily lead to memory configuration conflicts.
+
+### Task (Operator) Heap Memory
+
+If you want to guarantee that a certain amount of JVM heap is available for your user code, you can set the *task heap memory*
+explicitly ([taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size)).
+It will be added to the JVM heap size and will be dedicated to Flink’s operators running the user code.
+
+### Managed Memory
+
+*Managed memory* is managed by Flink and is allocated as native memory (off-heap). The following workloads use *managed memory*:
+* Streaming jobs can use it for [RocksDB state backend](../state/state_backends.html#the-rocksdbstatebackend).
+* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching of intermediate results.
+
+The size of *managed memory* can be
+* either configured explicitly via [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+* or computed as a fraction of *total Flink memory* via [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction)
+
+*size* will override *fraction*, if both are set.
+If neither *size* nor *fraction* is explicitly configured, the [default fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
+
+See also [how to configure memory for state backends](mem_tuning.html#configure-memory-for-state-backends) and [batch jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+
+## Configure Off-Heap Memory (direct or native)
+
+The off-heap memory which is allocated by user code should be accounted for in *task off-heap memory*
+([taskmanager.memory.task.off-heap.size](../config.html#taskmanager-memory-task-off-heap-size)).
+
+<strong>Note:</strong> You can also increase the [framework off-heap memory](#framework-memory). This option is advanced
+and only recommended to be changed if you are sure that the Flink framework needs more memory.
+
+Flink includes the *framework off-heap memory* and *task off-heap memory* into the *direct memory* limit of the JVM,
+see also [JVM parameters](#jvm-parameters).
+
+<strong>Note:</strong> Although, native non-direct memory usage can be accounted for as a part of the
+*framework off-heap memory* or *task off-heap memory*, it means that the JVM's *direct memory* limit will be higher.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+to never exceed its configured size. Therefore, resizing the *network memory* will not help in this situation.
+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model
+
+This section gives a detailed description of all components in Flink’s memory model.
+
+<br />
+<center>
+ <img src="{{ site.baseurl }}/fig/detailed-mem-model.svg" width="300px" alt="Simple memory model" usemap="#simple-mem-model">
+</center>
+<br />
+\* notice, that the *native non-direct* usage of memory in user code can be also accounted for as a part of the *task off-heap memory*,
+see also [how to configure off-heap memory](#configure-off-heap-memory-direct-or-native)
Review comment:
I can do this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r374478914
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,184 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)
+
+The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
+
+| **Removed option** | **Note** |
+| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
+| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
+
+The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:
+
+| **Deprecated option** | **Interpreted as** |
+| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
+| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
+| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
+| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
+| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
+
+Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
+It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
+See also [new detailed memory model](mem_setup.html#detailed-memory-model).
+
+The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
+have no effect for task manager processes anymore but they still have the same semantics for the job manager process.
+See also [how to migrate container cut-off](#container-cut-off-memory).
+
+## Total Memory (previously Heap Memory)
+
+The previous options which were responsible for the total memory used by Flink are ‘taskmanager.heap.size’ or ‘taskmanager.heap.mb’.
+Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.
+
+The Mesos integration also had a separate option with the same semantics: ‘mesos.resourcemanager.tasks.mem’ which has also been deprecated.
+
+If the mentioned legacy options are used without specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) for standalone deployments
Review comment:
```suggestion
* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size)) for standalone deployments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10999:
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide
URL: https://github.com/apache/flink/pull/10999#discussion_r376764162
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"
+nav-parent_id: ops_mem
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options
+were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one.
+
+* toc
+{:toc}
+
+<div class="alert alert-warning">
+ <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can
+ result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions
+ before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application.
+</div>
+
+<span class="label label-info">Note</span> The previous memory configuration allows that no memory related options are set at all
+as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
+that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
+* [`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
+* [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+* [`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size) and [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
+
+The [default `flink-conf.yaml`](#default-configuration-in-flink-confyaml) shipped with Flink sets [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+to make the default memory configuration consistent.
+
+This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
+to evaluate and compare the results of the legacy and new memory computations.
+
+## Changes in Configuration Options
+
+This chapter shortly lists all changes in the memory configuration options before the 1.10 release.
+It also references other chapters for more details about the migration to the new configuration options.
+
+The following options are completely removed. If they are still used, they will be just ignored.
Review comment:
```suggestion
The following options are completely removed. If they are still used, they will be ignored.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services