You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tr...@apache.org on 2017/02/12 15:23:45 UTC
flink git commit: [FLINK-5712] [config] update several deprecated
configuration options
Repository: flink
Updated Branches:
refs/heads/master ff3786663 -> 4f47ccdcb
[FLINK-5712] [config] update several deprecated configuration options
This closes #3267.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/4f47ccdc
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/4f47ccdc
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/4f47ccdc
Branch: refs/heads/master
Commit: 4f47ccdcbd3c794522d9bd5b60d11c016a71b5e7
Parents: ff37866
Author: fengyelei <fe...@gmail.com>
Authored: Sat Feb 4 17:18:06 2017 +0800
Committer: Till Rohrmann <tr...@apache.org>
Committed: Sun Feb 12 16:22:36 2017 +0100
----------------------------------------------------------------------
docs/setup/config.md | 12 ++++++------
docs/setup/mesos.md | 4 ++--
.../org/apache/flink/configuration/ConfigConstants.java | 6 +++++-
.../clusterframework/services/MesosServicesUtils.java | 7 +++++--
4 files changed, 18 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/4f47ccdc/docs/setup/config.md
----------------------------------------------------------------------
diff --git a/docs/setup/config.md b/docs/setup/config.md
index b86c534..5c912cd 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -404,9 +404,9 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use
### YARN
-- `yarn.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
+- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
-- `yarn.heap-cutoff-min`: (Default 384 MB) Minimum amount of memory to cut off the requested heap size.
+- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size.
- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure.
@@ -418,11 +418,11 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use
- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise.
-- `yarn.application-master.env.`*ENV_VAR1=value* Configuration values prefixed with `yarn.application-master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set:
+- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set:
- `yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"`
+ `containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"`
-- `yarn.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes.
+- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes.
- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN:
`%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%`. This configuration parameter allows users
@@ -474,7 +474,7 @@ May be set to -1 to disable this feature.
- `mesos.resourcemanager.tasks.container.image.name`: Image name to use for the container (**NO DEFAULT**)
-- `recovery.zookeeper.path.mesos-workers`: The ZooKeeper root path for persisting the Mesos worker information.
+- `high-availability.zookeeper.path.mesos-workers`: The ZooKeeper root path for persisting the Mesos worker information.
### High Availability (HA)
http://git-wip-us.apache.org/repos/asf/flink/blob/4f47ccdc/docs/setup/mesos.md
----------------------------------------------------------------------
diff --git a/docs/setup/mesos.md b/docs/setup/mesos.md
index 032477a..77267cb 100644
--- a/docs/setup/mesos.md
+++ b/docs/setup/mesos.md
@@ -217,7 +217,7 @@ For example:
You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
-For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set `high-availability.zookeeper.path.mesos-workers` to a valid Zookeeper path.
#### Marathon
@@ -257,7 +257,7 @@ May be set to -1 to disable this feature.
`mesos.resourcemanager.framework.role`: Mesos framework role definition (**DEFAULT:** *)
-`recovery.zookeeper.path.mesos-workers`: The ZooKeeper root path for persisting the Mesos worker information.
+`high-availability.zookeeper.path.mesos-workers`: The ZooKeeper root path for persisting the Mesos worker information.
`mesos.resourcemanager.framework.principal`: Mesos framework principal (**NO DEFAULT**)
http://git-wip-us.apache.org/repos/asf/flink/blob/4f47ccdc/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
----------------------------------------------------------------------
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
index c608fde..76c1cf7 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
@@ -820,7 +820,7 @@ public final class ConfigConstants {
/** ZooKeeper root path (ZNode) for Mesos workers. */
@PublicEvolving
- public static final String HA_ZOOKEEPER_MESOS_WORKERS_PATH = "recovery.zookeeper.path.mesos-workers";
+ public static final String HA_ZOOKEEPER_MESOS_WORKERS_PATH = "high-availability.zookeeper.path.mesos-workers";
@PublicEvolving
public static final String HA_ZOOKEEPER_SESSION_TIMEOUT = "high-availability.zookeeper.client.session-timeout";
@@ -879,6 +879,10 @@ public final class ConfigConstants {
@Deprecated
public static final String ZOOKEEPER_CHECKPOINT_COUNTER_PATH = "recovery.zookeeper.path.checkpoint-counter";
+ /** Deprecated in favour of {@link #HA_ZOOKEEPER_MESOS_WORKERS_PATH}. */
+ @Deprecated
+ public static final String ZOOKEEPER_MESOS_WORKERS_PATH = "recovery.zookeeper.path.mesos-workers";
+
/** Deprecated in favour of {@link #HA_ZOOKEEPER_SESSION_TIMEOUT}. */
@Deprecated
public static final String ZOOKEEPER_SESSION_TIMEOUT = "recovery.zookeeper.client.session-timeout";
http://git-wip-us.apache.org/repos/asf/flink/blob/4f47ccdc/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/services/MesosServicesUtils.java
----------------------------------------------------------------------
diff --git a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/services/MesosServicesUtils.java b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/services/MesosServicesUtils.java
index 13eb759..0ac5f4e 100644
--- a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/services/MesosServicesUtils.java
+++ b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/services/MesosServicesUtils.java
@@ -22,6 +22,7 @@ import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobmanager.HighAvailabilityMode;
import org.apache.flink.runtime.zookeeper.ZooKeeperUtilityFactory;
+import org.apache.flink.util.ConfigurationUtil;
public class MesosServicesUtils {
@@ -40,9 +41,11 @@ public class MesosServicesUtils {
return new StandaloneMesosServices();
case ZOOKEEPER:
- final String zkMesosRootPath = configuration.getString(
+ final String zkMesosRootPath = ConfigurationUtil.getStringWithDeprecatedKeys(
+ configuration,
ConfigConstants.HA_ZOOKEEPER_MESOS_WORKERS_PATH,
- ConfigConstants.DEFAULT_ZOOKEEPER_MESOS_WORKERS_PATH);
+ ConfigConstants.DEFAULT_ZOOKEEPER_MESOS_WORKERS_PATH,
+ ConfigConstants.ZOOKEEPER_MESOS_WORKERS_PATH);
ZooKeeperUtilityFactory zooKeeperUtilityFactory = new ZooKeeperUtilityFactory(
configuration,