You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by pn...@apache.org on 2022/06/21 15:46:18 UTC
[flink] branch master updated (f8f54bc1831 -> f88d276bb42)
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from f8f54bc1831 [FLINK-28044][runtime][security] Make hadoop filesystems configuration available to all deployment targets
new a17d824e179 [FLINK-26762][docs] Document overdraft buffers
new e374b7f1682 fixup! [FLINK-26762][docs] Document overdraft buffers
new f88d276bb42 fixup! fixup! [FLINK-26762][docs] Document overdraft buffers
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../docs/deployment/memory/network_mem_tuning.md | 17 +++++++++++++++++
.../docs/deployment/memory/network_mem_tuning.md | 17 +++++++++++++++++
2 files changed, 34 insertions(+)
[flink] 02/03: fixup! [FLINK-26762][docs] Document overdraft buffers
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit e374b7f16825e38d5de4be1cf307b9abbd5aab63
Author: Piotr Nowojski <pi...@gmail.com>
AuthorDate: Tue Jun 21 14:30:14 2022 +0200
fixup! [FLINK-26762][docs] Document overdraft buffers
---
.../docs/deployment/memory/network_mem_tuning.md | 20 ++++++++++++--------
.../docs/deployment/memory/network_mem_tuning.md | 20 ++++++++++++--------
2 files changed, 24 insertions(+), 16 deletions(-)
diff --git a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
index 8adc281e225..39527284814 100644
--- a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
@@ -120,16 +120,20 @@ Flink 有多个本地缓冲区池 —— 每个输出和输入流对应一个。
#### Overdraft buffers
-For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers.
-Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output.
-This can happen in situations like:
+For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate`
+(by default 5) extra overdraft buffers. Those buffers are only used, if the subtask is backpressured
+by downstream subtasks but the subtask can not gracefully pause its current process. This can happen
+in situations like:
- Serializing very large records, that do not fit into a single network buffer.
- Flat Map like operator, that produces many output records per single input record.
-- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`).
-
-Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints
-from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can
-make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers.
+- Operators that output many records either periodically or on a reaction to some events (for
+example `WindowOperator`'s triggers).
+
+Without overdraft buffers in such situations Flink subtask thread would block on the backpressure,
+preventing for example unaligned checkpoints from completing. To mitigate this, the overdraft
+buffers concept has been added. Those overdraft buffers are strictly optional and Flink can
+gradually make progress using only regular buffers, which means `0` is an acceptable configuration
+for the `taskmanager.network.memory.max-overdraft-buffers-per-gate`.
## 缓冲区的数量
diff --git a/docs/content/docs/deployment/memory/network_mem_tuning.md b/docs/content/docs/deployment/memory/network_mem_tuning.md
index a29cdfa133d..00f8159e016 100644
--- a/docs/content/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content/docs/deployment/memory/network_mem_tuning.md
@@ -122,16 +122,20 @@ Unlike the input buffer pool, the configured amount of exclusive buffers and flo
#### Overdraft buffers
-For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers.
-Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output.
-This can happen in situations like:
+For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate`
+(by default 5) extra overdraft buffers. Those buffers are only used, if the subtask is backpressured
+by downstream subtasks but the subtask can not gracefully pause its current process. This can happen
+in situations like:
- Serializing very large records, that do not fit into a single network buffer.
- Flat Map like operator, that produces many output records per single input record.
-- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`).
-
-Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints
-from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can
-make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers.
+- Operators that output many records either periodically or on a reaction to some events (for
+example `WindowOperator`'s triggers).
+
+Without overdraft buffers in such situations Flink subtask thread would block on the backpressure,
+preventing for example unaligned checkpoints from completing. To mitigate this, the overdraft
+buffers concept has been added. Those overdraft buffers are strictly optional and Flink can
+gradually make progress using only regular buffers, which means `0` is an acceptable configuration
+for the `taskmanager.network.memory.max-overdraft-buffers-per-gate`.
## The number of in-flight buffers
[flink] 03/03: fixup! fixup! [FLINK-26762][docs] Document overdraft buffers
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit f88d276bb4271cc5f1f52b6af68a5a7250f67c89
Author: Piotr Nowojski <pi...@gmail.com>
AuthorDate: Tue Jun 21 17:12:07 2022 +0200
fixup! fixup! [FLINK-26762][docs] Document overdraft buffers
---
docs/content.zh/docs/deployment/memory/network_mem_tuning.md | 4 ++--
docs/content/docs/deployment/memory/network_mem_tuning.md | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
index 39527284814..b7e49d3218b 100644
--- a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
@@ -122,8 +122,8 @@ Flink 有多个本地缓冲区池 —— 每个输出和输入流对应一个。
For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate`
(by default 5) extra overdraft buffers. Those buffers are only used, if the subtask is backpressured
-by downstream subtasks but the subtask can not gracefully pause its current process. This can happen
-in situations like:
+by downstream subtasks and the subtask requires more than a single network buffer to finish what its
+currently doing. This can happen in situations like:
- Serializing very large records, that do not fit into a single network buffer.
- Flat Map like operator, that produces many output records per single input record.
- Operators that output many records either periodically or on a reaction to some events (for
diff --git a/docs/content/docs/deployment/memory/network_mem_tuning.md b/docs/content/docs/deployment/memory/network_mem_tuning.md
index 00f8159e016..032fd3f5a17 100644
--- a/docs/content/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content/docs/deployment/memory/network_mem_tuning.md
@@ -124,8 +124,8 @@ Unlike the input buffer pool, the configured amount of exclusive buffers and flo
For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate`
(by default 5) extra overdraft buffers. Those buffers are only used, if the subtask is backpressured
-by downstream subtasks but the subtask can not gracefully pause its current process. This can happen
-in situations like:
+by downstream subtasks and the subtask requires more than a single network buffer to finish what its
+currently doing. This can happen in situations like:
- Serializing very large records, that do not fit into a single network buffer.
- Flat Map like operator, that produces many output records per single input record.
- Operators that output many records either periodically or on a reaction to some events (for
[flink] 01/03: [FLINK-26762][docs] Document overdraft buffers
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a17d824e1797f3424cea9c12e0a9bc9a4d654741
Author: Piotr Nowojski <pi...@gmail.com>
AuthorDate: Tue Jun 21 10:14:09 2022 +0200
[FLINK-26762][docs] Document overdraft buffers
---
.../content.zh/docs/deployment/memory/network_mem_tuning.md | 13 +++++++++++++
docs/content/docs/deployment/memory/network_mem_tuning.md | 13 +++++++++++++
2 files changed, 26 insertions(+)
diff --git a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
index cccba3bff4b..8adc281e225 100644
--- a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md
@@ -118,6 +118,19 @@ Flink 有多个本地缓冲区池 —— 每个输出和输入流对应一个。
不同于输入缓冲区池，这里配置的独占缓冲区和流动缓冲区只被当作推荐值。如果没有足够的缓冲区，每个输出 subpartition 可以只使用一个独占缓冲区而没有流动缓冲区。
+#### Overdraft buffers
+
+For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers.
+Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output.
+This can happen in situations like:
+- Serializing very large records, that do not fit into a single network buffer.
+- Flat Map like operator, that produces many output records per single input record.
+- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`).
+
+Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints
+from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can
+make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers.
+
## 缓冲区的数量
独占缓冲区和流动缓冲区的默认配置应该足以应对最大吞吐。如果想要最小化缓冲数据量，那么可以将独占缓冲区设置为 `0`，同时减小内存段的大小。
diff --git a/docs/content/docs/deployment/memory/network_mem_tuning.md b/docs/content/docs/deployment/memory/network_mem_tuning.md
index 9c537517b89..a29cdfa133d 100644
--- a/docs/content/docs/deployment/memory/network_mem_tuning.md
+++ b/docs/content/docs/deployment/memory/network_mem_tuning.md
@@ -120,6 +120,19 @@ In order to avoid excessive data skew, the number of buffers for each subpartiti
Unlike the input buffer pool, the configured amount of exclusive buffers and floating buffers is only treated as recommended values. If there are not enough buffers available, Flink can make progress with only a single exclusive buffer per output subpartition and zero floating buffers.
+#### Overdraft buffers
+
+For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers.
+Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output.
+This can happen in situations like:
+- Serializing very large records, that do not fit into a single network buffer.
+- Flat Map like operator, that produces many output records per single input record.
+- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`).
+
+Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints
+from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can
+make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers.
+
## The number of in-flight buffers
The default settings for exclusive buffers and floating buffers should be sufficient for the maximum throughput. If the minimum of in-flight data needs to be set, the exclusive buffers can be set to `0` and the memory segment size can be decreased.