You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/08/31 10:43:23 UTC
[GitHub] [flink-web] XComp commented on a change in pull request #373: Add blog post: Memory Management improvements for Flink’s JobManager in Apache Flink 1.11
XComp commented on a change in pull request #373:
URL: https://github.com/apache/flink-web/pull/373#discussion_r479997267
##########
File path: _posts/2020-09-01-flink-1.11-memory-management-improvements.md
##########
@@ -0,0 +1,66 @@
+---
+layout: post
+title: "Memory Management improvements for Flink’s JobManager in Apache Flink 1.11"
+date: 2020-09-01T15:30:00.000Z
+authors:
+- Andrey:
+ name: "Andrey Zagrebin"
+categories: news
+excerpt: In a previous blog post focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead.
+---
+
+Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager and configuration options for your Flink clusters. These recently-introduced changes make Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing better control over its memory consumption.
+
+The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html), focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink 1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of the JobManager follows a similar approach to that of the TaskManagers; it is simpler and has fewer components and tuning knobs. This post might then seem very similar to our previous story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager memory model as of Flink 1.11. Read on for a full list of updates and changes below!
+
+## Introduction to Flink’s process memory model
+
+Having a clear understanding of Apache Flink’s process memory model allows you to manage resources for the various workloads more efficiently. The following diagram illustrates the main memory components of a Flink process:
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-09-01-flink-1.11-memory-management-improvements/total-process-memory-flink-1.11.png" width="400px" alt="Backpressure sampling:high"/>
+<br/>
+<i><small>Flink: Total Process Memory</small></i>
+</center>
+<br/>
+
+The JobManager process is a JVM process. On a high level, its memory consists of the JVM Heap and Off-Heap memory. These types of memory are consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace etc). There are two major memory consumers within the JobManager process: the framework itself consuming memory for internal data structures, network communication, etc. and the user code which runs within the JobManager process, e.g. in certain batch sources or in checkpoint completion callbacks.
Review comment:
```suggestion
The JobManager process is a JVM process. On a high level, its memory consists of the JVM Heap and Off-Heap memory. These types of memory are consumed by Flink directly or by the JVM for its specific purposes (i.e. metaspace etc). There are two major memory consumers within the JobManager process: the framework itself consuming memory for internal data structures, network communication, etc. and the user code which runs within the JobManager process, e.g. in certain batch sources or in checkpoint completion callbacks.
```
##########
File path: _posts/2020-09-01-flink-1.11-memory-management-improvements.md
##########
@@ -0,0 +1,66 @@
+---
+layout: post
+title: "Memory Management improvements for Flink’s JobManager in Apache Flink 1.11"
+date: 2020-09-01T15:30:00.000Z
+authors:
+- Andrey:
+ name: "Andrey Zagrebin"
+categories: news
+excerpt: In a previous blog post focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead.
+---
+
+Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager and configuration options for your Flink clusters. These recently-introduced changes make Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing better control over its memory consumption.
+
+The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html), focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink 1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of the JobManager follows a similar approach to that of the TaskManagers; it is simpler and has fewer components and tuning knobs. This post might then seem very similar to our previous story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager memory model as of Flink 1.11. Read on for a full list of updates and changes below!
Review comment:
```suggestion
The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html) focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink 1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of the JobManager follows a similar approach to that of the TaskManagers; it is simpler and has fewer components and tuning knobs. This post might seem very similar to our previous story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager memory model as of Flink 1.11. Read on for a full list of updates and changes below!
```
##########
File path: _posts/2020-09-01-flink-1.11-memory-management-improvements.md
##########
@@ -0,0 +1,66 @@
+---
+layout: post
+title: "Memory Management improvements for Flink’s JobManager in Apache Flink 1.11"
+date: 2020-09-01T15:30:00.000Z
+authors:
+- Andrey:
+ name: "Andrey Zagrebin"
+categories: news
+excerpt: In a previous blog post focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead.
+---
+
+Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager and configuration options for your Flink clusters. These recently-introduced changes make Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing better control over its memory consumption.
+
+The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html), focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink 1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of the JobManager follows a similar approach to that of the TaskManagers; it is simpler and has fewer components and tuning knobs. This post might then seem very similar to our previous story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager memory model as of Flink 1.11. Read on for a full list of updates and changes below!
Review comment:
Ok, it's hard to see in the diff above: I removed the comma in the first sentence of this paragraph.
##########
File path: _posts/2020-09-01-flink-1.11-memory-management-improvements.md
##########
@@ -0,0 +1,66 @@
+---
+layout: post
+title: "Memory Management improvements for Flink’s JobManager in Apache Flink 1.11"
+date: 2020-09-01T15:30:00.000Z
+authors:
+- Andrey:
+ name: "Andrey Zagrebin"
+categories: news
+excerpt: In a previous blog post focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead.
+---
+
+Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager and configuration options for your Flink clusters. These recently-introduced changes make Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing better control over its memory consumption.
+
+The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html), focused on the memory model of the TaskManagers and how it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink 1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of the JobManager follows a similar approach to that of the TaskManagers; it is simpler and has fewer components and tuning knobs. This post might then seem very similar to our previous story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager memory model as of Flink 1.11. Read on for a full list of updates and changes below!
+
+## Introduction to Flink’s process memory model
+
+Having a clear understanding of Apache Flink’s process memory model allows you to manage resources for the various workloads more efficiently. The following diagram illustrates the main memory components of a Flink process:
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-09-01-flink-1.11-memory-management-improvements/total-process-memory-flink-1.11.png" width="400px" alt="Backpressure sampling:high"/>
+<br/>
+<i><small>Flink: Total Process Memory</small></i>
+</center>
+<br/>
+
+The JobManager process is a JVM process. On a high level, its memory consists of the JVM Heap and Off-Heap memory. These types of memory are consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace etc). There are two major memory consumers within the JobManager process: the framework itself consuming memory for internal data structures, network communication, etc. and the user code which runs within the JobManager process, e.g. in certain batch sources or in checkpoint completion callbacks.
Review comment:
I don't see the difference, again. -.- Hence, here's what I changed: I added `the` in the second sentence: "These types of memory are consumed by Flink directly or by ***the*** JVM for [...]"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org