You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fc...@apache.org on 2025/01/23 17:34:23 UTC
(flink) branch release-1.20 updated (d46e1ada621 -> 72a32327c1b)
This is an automated email from the ASF dual-hosted git repository.
fcsaky pushed a change to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
from d46e1ada621 [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4
new 976c77db658 Revert "[FLINK-36348][test] Fix the Netty shuffle direct memory test by increase the direct memory of TM"
new 72a32327c1b [FLINK-37100][tests] Fix `test_netty_shuffle_memory_control.sh` with Netty4 RPC
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../test-scripts/test_netty_shuffle_memory_control.sh | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
(flink) 01/02: Revert "[FLINK-36348][test] Fix the Netty shuffle direct memory test by increase the direct memory of TM"
Posted by fc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fcsaky pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 976c77db658bad3f452696fdeb8422a1ea3c93e0
Author: Ferenc Csaky <fc...@apache.org>
AuthorDate: Thu Jan 23 17:17:26 2025 +0100
Revert "[FLINK-36348][test] Fix the Netty shuffle direct memory test by increase the direct memory of TM"
This reverts commit 407c3d53e8b1dcaae1bf82598227a47b0340f650.
---
.../test-scripts/test_netty_shuffle_memory_control.sh | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh b/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
index cad60f2d9b4..71be5e1cbca 100755
--- a/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
+++ b/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
@@ -23,15 +23,18 @@ TEST=flink-netty-shuffle-memory-control-test
TEST_PROGRAM_NAME=NettyShuffleMemoryControlTestProgram
TEST_PROGRAM_JAR=${END_TO_END_DIR}/$TEST/target/$TEST_PROGRAM_NAME.jar
-set_config_key "taskmanager.memory.flink.size" "600m"
+set_config_key "taskmanager.memory.flink.size" "512m"
set_config_key "taskmanager.memory.network.min" "128m"
set_config_key "taskmanager.memory.network.max" "128m"
# 20 slots per task manager.
set_config_key "taskmanager.numberOfTaskSlots" "20"
-# Limits the direct memory to be one chunk (4M * 20) plus some margins.
-set_config_key "taskmanager.memory.framework.off-heap.size" "90m"
+# Sets only one arena per TM for boosting the netty internal memory overhead.
+set_config_key "taskmanager.network.netty.num-arenas" "1"
+
+# Limits the direct memory to be one chunk (4M) plus some margins.
+set_config_key "taskmanager.memory.framework.off-heap.size" "7m"
# Starts the cluster which includes one TaskManager.
start_cluster
(flink) 02/02: [FLINK-37100][tests] Fix `test_netty_shuffle_memory_control.sh` with Netty4 RPC
Posted by fc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fcsaky pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 72a32327c1bef284b3509323cef8e12540efd479
Author: Ferenc Csaky <fc...@apache.org>
AuthorDate: Thu Jan 23 17:18:07 2025 +0100
[FLINK-37100][tests] Fix `test_netty_shuffle_memory_control.sh` with Netty4 RPC
With Pekko updated and using Netty4, the default memory buffer allocation
is different compared to Netty3, thus to stabilize this test we increased
the given memory a bit.
---
.../test-scripts/test_netty_shuffle_memory_control.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh b/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
index 71be5e1cbca..d5331b2071f 100755
--- a/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
+++ b/flink-end-to-end-tests/test-scripts/test_netty_shuffle_memory_control.sh
@@ -34,7 +34,7 @@ set_config_key "taskmanager.numberOfTaskSlots" "20"
set_config_key "taskmanager.network.netty.num-arenas" "1"
# Limits the direct memory to be one chunk (4M) plus some margins.
-set_config_key "taskmanager.memory.framework.off-heap.size" "7m"
+set_config_key "taskmanager.memory.framework.off-heap.size" "12m"
# Starts the cluster which includes one TaskManager.
start_cluster