You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ch...@apache.org on 2019/07/16 10:37:49 UTC
[flink] branch master updated: [FLINK-11860][config] Update usages
of deprecated (job-/taskmanager).heap.mb
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 9cdae73 [FLINK-11860][config] Update usages of deprecated (job-/taskmanager).heap.mb
9cdae73 is described below
commit 9cdae7307e9be42be473a22d5a3e924c159ca12d
Author: Yun Tang <my...@live.com>
AuthorDate: Fri Mar 15 00:15:05 2019 +0800
[FLINK-11860][config] Update usages of deprecated (job-/taskmanager).heap.mb
---
docs/ops/config.md | 2 +-
docs/ops/deployment/cluster_setup.md | 6 +++---
docs/ops/deployment/cluster_setup.zh.md | 6 +++---
docs/ops/deployment/mesos.md | 6 +++---
docs/ops/deployment/mesos.zh.md | 6 +++---
flink-end-to-end-tests/test-scripts/common.sh | 4 ++--
flink-end-to-end-tests/test-scripts/test_heavy_deployment.sh | 2 +-
.../test-scripts/test_high_parallelism_iterations.sh | 2 +-
flink-jepsen/src/jepsen/flink/db.clj | 4 ++--
9 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/docs/ops/config.md b/docs/ops/config.md
index 6a444a0..ffc6049 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -70,7 +70,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp
{% include generated/task_manager_configuration.html %}
-For *batch* jobs (or if `taskmanager.memoy.preallocate` is enabled) Flink allocates a fraction of 0.7 of the free memory (total memory configured via taskmanager.heap.mb minus memory used for network buffers) for its managed memory. Managed memory helps Flink to run the batch operators efficiently. It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations. If Flink runs out of managed memory, it utilizes disk space. Using managed memory, some [...]
+For *batch* jobs (or if `taskmanager.memoy.preallocate` is enabled) Flink allocates a fraction of 0.7 of the free memory (total memory configured via taskmanager.heap.size minus memory used for network buffers) for its managed memory. Managed memory helps Flink to run the batch operators efficiently. It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations. If Flink runs out of managed memory, it utilizes disk space. Using managed memory, som [...]
The default fraction for managed memory can be adjusted using the taskmanager.memory.fraction parameter. An absolute value may be set using taskmanager.memory.size (overrides the fraction parameter). If desired, the managed memory may be allocated outside the JVM heap. This may improve performance in setups with large memory sizes.
diff --git a/docs/ops/deployment/cluster_setup.md b/docs/ops/deployment/cluster_setup.md
index bcade78..003ef3c 100644
--- a/docs/ops/deployment/cluster_setup.md
+++ b/docs/ops/deployment/cluster_setup.md
@@ -68,7 +68,7 @@ cd flink-*
After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.mb` and `taskmanager.heap.mb` keys.
+Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.size` and `taskmanager.heap.size` keys.
These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting the environment variable `FLINK_TM_HEAP` on those specific nodes.
@@ -106,8 +106,8 @@ Please see the [configuration page](../config.html) for details and additional c
In particular,
- * the amount of available memory per JobManager (`jobmanager.heap.mb`),
- * the amount of available memory per TaskManager (`taskmanager.heap.mb`),
+ * the amount of available memory per JobManager (`jobmanager.heap.size`),
+ * the amount of available memory per TaskManager (`taskmanager.heap.size`),
* the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
* the total number of CPUs in the cluster (`parallelism.default`) and
* the temporary directories (`io.tmp.dirs`)
diff --git a/docs/ops/deployment/cluster_setup.zh.md b/docs/ops/deployment/cluster_setup.zh.md
index 506c66f..812d1b1 100644
--- a/docs/ops/deployment/cluster_setup.zh.md
+++ b/docs/ops/deployment/cluster_setup.zh.md
@@ -68,7 +68,7 @@ cd flink-*
After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.mb` and `taskmanager.heap.mb` keys.
+Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.size` and `taskmanager.heap.size` keys.
These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting the environment variable `FLINK_TM_HEAP` on those specific nodes.
@@ -106,8 +106,8 @@ Please see the [configuration page](../config.html) for details and additional c
In particular,
- * the amount of available memory per JobManager (`jobmanager.heap.mb`),
- * the amount of available memory per TaskManager (`taskmanager.heap.mb`),
+ * the amount of available memory per JobManager (`jobmanager.heap.size`),
+ * the amount of available memory per TaskManager (`taskmanager.heap.size`),
* the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
* the total number of CPUs in the cluster (`parallelism.default`) and
* the temporary directories (`io.tmp.dirs`)
diff --git a/docs/ops/deployment/mesos.md b/docs/ops/deployment/mesos.md
index 8395bd0..476e16e 100644
--- a/docs/ops/deployment/mesos.md
+++ b/docs/ops/deployment/mesos.md
@@ -214,11 +214,11 @@ For example:
bin/mesos-appmaster.sh \
-Dmesos.master=master.foobar.org:5050 \
- -Djobmanager.heap.mb=1024 \
+ -Djobmanager.heap.size=1024m \
-Djobmanager.rpc.port=6123 \
-Drest.port=8081 \
-Dmesos.resourcemanager.tasks.mem=4096 \
- -Dtaskmanager.heap.mb=3500 \
+ -Dtaskmanager.heap.size=3500m \
-Dtaskmanager.numberOfTaskSlots=2 \
-Dparallelism.default=10
@@ -236,7 +236,7 @@ Here is an example configuration for Marathon:
{
"id": "flink",
- "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
+ "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.size=1024m -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.size=1024m -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
"cpus": 1.0,
"mem": 1024
}
diff --git a/docs/ops/deployment/mesos.zh.md b/docs/ops/deployment/mesos.zh.md
index f58fb32..88874de 100644
--- a/docs/ops/deployment/mesos.zh.md
+++ b/docs/ops/deployment/mesos.zh.md
@@ -214,11 +214,11 @@ For example:
bin/mesos-appmaster.sh \
-Dmesos.master=master.foobar.org:5050 \
- -Djobmanager.heap.mb=1024 \
+ -Djobmanager.heap.size=1024m \
-Djobmanager.rpc.port=6123 \
-Drest.port=8081 \
-Dmesos.resourcemanager.tasks.mem=4096 \
- -Dtaskmanager.heap.mb=3500 \
+ -Dtaskmanager.heap.size=3500m \
-Dtaskmanager.numberOfTaskSlots=2 \
-Dparallelism.default=10
@@ -236,7 +236,7 @@ Here is an example configuration for Marathon:
{
"id": "flink",
- "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
+ "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.size=1024m -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
"cpus": 1.0,
"mem": 1024
}
diff --git a/flink-end-to-end-tests/test-scripts/common.sh b/flink-end-to-end-tests/test-scripts/common.sh
index 4eac9f4..a4fff94 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -156,8 +156,8 @@ function create_ha_config() {
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
- jobmanager.heap.mb: 1024
- taskmanager.heap.mb: 1024
+ jobmanager.heap.size: 1024m
+ taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: ${TASK_SLOTS_PER_TM_HA}
#==============================================================================
diff --git a/flink-end-to-end-tests/test-scripts/test_heavy_deployment.sh b/flink-end-to-end-tests/test-scripts/test_heavy_deployment.sh
index f82cd3f..06ad114 100755
--- a/flink-end-to-end-tests/test-scripts/test_heavy_deployment.sh
+++ b/flink-end-to-end-tests/test-scripts/test_heavy_deployment.sh
@@ -25,7 +25,7 @@ TEST=flink-heavy-deployment-stress-test
TEST_PROGRAM_NAME=HeavyDeploymentStressTestProgram
TEST_PROGRAM_JAR=${END_TO_END_DIR}/$TEST/target/$TEST_PROGRAM_NAME.jar
-set_config_key "taskmanager.heap.mb" "512" # 512Mb x 10TMs = 5Gb total heap
+set_config_key "taskmanager.heap.size" "512m" # 512Mb x 10TMs = 5Gb total heap
set_config_key "taskmanager.memory.size" "8" # 8Mb
set_config_key "taskmanager.network.memory.min" "8mb"
diff --git a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
index 2d2aec0..cfa5cae 100755
--- a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
+++ b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
@@ -25,7 +25,7 @@ TEST=flink-high-parallelism-iterations-test
TEST_PROGRAM_NAME=HighParallelismIterationsTestProgram
TEST_PROGRAM_JAR=${END_TO_END_DIR}/$TEST/target/$TEST_PROGRAM_NAME.jar
-set_config_key "taskmanager.heap.mb" "52" # 52Mb x 100 TMs = 5Gb total heap
+set_config_key "taskmanager.heap.size" "52m" # 52Mb x 100 TMs = 5Gb total heap
set_config_key "taskmanager.memory.size" "8" # 8Mb
set_config_key "taskmanager.network.memory.min" "8mb"
diff --git a/flink-jepsen/src/jepsen/flink/db.clj b/flink-jepsen/src/jepsen/flink/db.clj
index c636fcc..4aaa3f2 100644
--- a/flink-jepsen/src/jepsen/flink/db.clj
+++ b/flink-jepsen/src/jepsen/flink/db.clj
@@ -296,10 +296,10 @@
(str install-dir "/bin/mesos-appmaster.sh")
(str "-Dmesos.master=" (zookeeper-uri test mesos/zk-namespace))
"-Djobmanager.rpc.address=$(hostname -f)"
- "-Djobmanager.heap.mb=2048"
+ "-Djobmanager.heap.size=2048m"
"-Djobmanager.rpc.port=6123"
"-Dmesos.resourcemanager.tasks.mem=2048"
- "-Dtaskmanager.heap.mb=2048"
+ "-Dtaskmanager.heap.size=2048m"
"-Dmesos.resourcemanager.tasks.cpus=1"
"-Drest.bind-address=$(hostname -f)"))