You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2018/08/08 11:33:38 UTC
[GitHub] dawidwys closed pull request #6444: [FLINK-9933] Simplify
taskmanager memory default values
dawidwys closed pull request #6444: [FLINK-9933] Simplify taskmanager memory default values
URL: https://github.com/apache/flink/pull/6444
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/_includes/generated/task_manager_configuration.html b/docs/_includes/generated/task_manager_configuration.html
index c18c5d4c36d..16c5d98f8be 100644
--- a/docs/_includes/generated/task_manager_configuration.html
+++ b/docs/_includes/generated/task_manager_configuration.html
@@ -84,7 +84,7 @@
</tr>
<tr>
<td><h5>taskmanager.memory.segment-size</h5></td>
- <td style="word-wrap: break-word;">"32768"</td>
+ <td style="word-wrap: break-word;">"32kb"</td>
<td>Size of memory buffers used by the network stack and the memory manager.</td>
</tr>
<tr>
@@ -114,12 +114,12 @@
</tr>
<tr>
<td><h5>taskmanager.network.memory.max</h5></td>
- <td style="word-wrap: break-word;">"1073741824"</td>
+ <td style="word-wrap: break-word;">"1gb"</td>
<td>Maximum memory size for network buffers.</td>
</tr>
<tr>
<td><h5>taskmanager.network.memory.min</h5></td>
- <td style="word-wrap: break-word;">"67108864"</td>
+ <td style="word-wrap: break-word;">"64mb"</td>
<td>Minimum memory size for network buffers.</td>
</tr>
<tr>
diff --git a/docs/ops/config.md b/docs/ops/config.md
index fd0df0c2370..d370e56d695 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -211,10 +211,10 @@ Previously, the number of network buffers was set manually which became a quite
network buffers with the following configuration parameters:
- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers (DEFAULT: 0.1),
-- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: 64 MB),
-- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: 1 GB), and
+- `taskmanager.network.memory.min`: Minimum memory size for network buffers (DEFAULT: 64MB),
+- `taskmanager.network.memory.max`: Maximum memory size for network buffers (DEFAULT: 1GB), and
- `taskmanager.memory.segment-size`: Size of memory buffers used by the memory manager and the
-network stack in bytes (DEFAULT: 32768 (= 32 KiBytes)).
+network stack in bytes (DEFAULT: 32KB).
#### Setting the Number of Network Buffers directly
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
index be76cf8c4db..98fa47f6697 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
@@ -190,11 +190,11 @@
// ------------------------------------------------------------------------
/**
- * Size of memory buffers used by the network stack and the memory manager (in bytes).
+ * Size of memory buffers used by the network stack and the memory manager.
*/
public static final ConfigOption<String> MEMORY_SEGMENT_SIZE =
key("taskmanager.memory.segment-size")
- .defaultValue("32768")
+ .defaultValue("32kb")
.withDescription("Size of memory buffers used by the network stack and the memory manager.");
/**
@@ -268,19 +268,19 @@
"` and \"taskmanager.network.memory.max\" may override this fraction.");
/**
- * Minimum memory size for network buffers (in bytes).
+ * Minimum memory size for network buffers.
*/
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MIN =
key("taskmanager.network.memory.min")
- .defaultValue(String.valueOf(64L << 20)) // 64 MB
+ .defaultValue("64mb")
.withDescription("Minimum memory size for network buffers.");
/**
- * Maximum memory size for network buffers (in bytes).
+ * Maximum memory size for network buffers.
*/
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MAX =
key("taskmanager.network.memory.max")
- .defaultValue(String.valueOf(1024L << 20)) // 1 GB
+ .defaultValue("1gb")
.withDescription("Maximum memory size for network buffers.");
/**
diff --git a/flink-dist/src/main/resources/flink-conf.yaml b/flink-dist/src/main/resources/flink-conf.yaml
index 62ed13dd81a..f4ec6a7e424 100644
--- a/flink-dist/src/main/resources/flink-conf.yaml
+++ b/flink-dist/src/main/resources/flink-conf.yaml
@@ -184,8 +184,8 @@ rest.port: 8081
# of network buffers" error. The default min is 64MB, teh default max is 1GB.
#
# taskmanager.network.memory.fraction: 0.1
-# taskmanager.network.memory.min: 67108864
-# taskmanager.network.memory.max: 1073741824
+# taskmanager.network.memory.min: 64mb
+# taskmanager.network.memory.max: 1gb
#==============================================================================
# Flink Cluster Security Configuration
diff --git a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
index 5a06266da64..3b684b664ba 100755
--- a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
+++ b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
@@ -30,9 +30,9 @@ backup_config
set_conf "taskmanager.heap.mb" "52" # 52Mb x 100 TMs = 5Gb total heap
set_conf "taskmanager.memory.size" "8" # 8Mb
-set_conf "taskmanager.network.memory.min" "8388608" # 8Mb
-set_conf "taskmanager.network.memory.max" "8388608" # 8Mb
-set_conf "taskmanager.memory.segment-size" "8192" # 8Kb
+set_conf "taskmanager.network.memory.min" "8mb"
+set_conf "taskmanager.network.memory.max" "8mb"
+set_conf "taskmanager.memory.segment-size" "8kb"
set_conf "taskmanager.network.netty.server.numThreads" "1"
set_conf "taskmanager.network.netty.client.numThreads" "1"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services