You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by al...@apache.org on 2017/07/24 16:37:36 UTC
[1/3] flink git commit: [FLINK-7241] Reword description of cluster-id
in HA doc
Repository: flink
Updated Branches:
refs/heads/master c0bad3b80 -> 640cc4ee6
[FLINK-7241] Reword description of cluster-id in HA doc
Before, we were describing what happens if you manually specify a
cluster-id. Now, we say explicitly that you should not do this when
running Flink on a resource manager.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/640cc4ee
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/640cc4ee
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/640cc4ee
Branch: refs/heads/master
Commit: 640cc4ee600d83cdcadabbf4193ad016d36bf4c8
Parents: d628e7a
Author: Aljoscha Krettek <al...@gmail.com>
Authored: Mon Jul 24 17:41:33 2017 +0200
Committer: Aljoscha Krettek <al...@gmail.com>
Committed: Mon Jul 24 18:36:48 2017 +0200
----------------------------------------------------------------------
docs/setup/jobmanager_high_availability.md | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/640cc4ee/docs/setup/jobmanager_high_availability.md
----------------------------------------------------------------------
diff --git a/docs/setup/jobmanager_high_availability.md b/docs/setup/jobmanager_high_availability.md
index 8754537..cb78c56 100644
--- a/docs/setup/jobmanager_high_availability.md
+++ b/docs/setup/jobmanager_high_availability.md
@@ -82,7 +82,13 @@ In order to start an HA-cluster add the following configuration keys to `conf/fl
<pre>high-availability.zookeeper.path.cluster-id: /default_ns # important: customize per cluster</pre>
- **Important**: if you are running multiple Flink HA clusters, you have to manually configure separate cluster-ids for each cluster. By default, the Yarn cluster and the Yarn session automatically generate cluster-ids based on Yarn application id. A manual configuration overrides this behaviour in Yarn. Specifying a cluster-id with the -z CLI option, in turn, overrides manual configuration.
+ **Important**: You should not set this value manually when runnig a YARN
+ cluster, a per-job YARN session, or on another cluster manager. In those
+ cases a cluster-id is automatically being generated based on the application
+ id. Manually setting a cluster-id overrides this behaviour in YARN.
+ Specifying a cluster-id with the -z CLI option, in turn, overrides manual
+ configuration. If you are running multiple Flink HA clusters on bare metal,
+ you have to manually configure separate cluster-ids for each cluster.
- **Storage directory** (required): JobManager metadata is persisted in the file system *storageDir* and only a pointer to this state is stored in ZooKeeper.
[3/3] flink git commit: [FLINK-7241] Rename namespace to cluster-id
in HA doc
Posted by al...@apache.org.
[FLINK-7241] Rename namespace to cluster-id in HA doc
This was renamed a while ago and namespace is now deprecated.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a671035f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a671035f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a671035f
Branch: refs/heads/master
Commit: a671035fa4e52cdf3540fa865473aa9404a841c4
Parents: c0bad3b
Author: Aljoscha Krettek <al...@gmail.com>
Authored: Fri Jul 21 17:45:24 2017 +0200
Committer: Aljoscha Krettek <al...@gmail.com>
Committed: Mon Jul 24 18:36:48 2017 +0200
----------------------------------------------------------------------
docs/setup/jobmanager_high_availability.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/a671035f/docs/setup/jobmanager_high_availability.md
----------------------------------------------------------------------
diff --git a/docs/setup/jobmanager_high_availability.md b/docs/setup/jobmanager_high_availability.md
index 5949835..e770fa8 100644
--- a/docs/setup/jobmanager_high_availability.md
+++ b/docs/setup/jobmanager_high_availability.md
@@ -74,15 +74,15 @@ In order to start an HA-cluster add the following configuration keys to `conf/fl
Each *addressX:port* refers to a ZooKeeper server, which is reachable by Flink at the given address and port.
-- **ZooKeeper root** (recommended): The *root ZooKeeper node*, under which all cluster namespace nodes are placed.
+- **ZooKeeper root** (recommended): The *root ZooKeeper node*, under which all cluster nodes are placed.
<pre>high-availability.zookeeper.path.root: /flink
-- **ZooKeeper namespace** (recommended): The *namespace ZooKeeper node*, under which all required coordination data for a cluster is placed.
+- **ZooKeeper cluster-id** (recommended): The *cluster-id ZooKeeper node*, under which all required coordination data for a cluster is placed.
- <pre>high-availability.zookeeper.path.namespace: /default_ns # important: customize per cluster</pre>
+ <pre>high-availability.zookeeper.path.cluster-id: /default_ns # important: customize per cluster</pre>
- **Important**: if you are running multiple Flink HA clusters, you have to manually configure separate namespaces for each cluster. By default, the Yarn cluster and the Yarn session automatically generate namespaces based on Yarn application id. A manual configuration overrides this behaviour in Yarn. Specifying a namespace with the -z CLI option, in turn, overrides manual configuration.
+ **Important**: if you are running multiple Flink HA clusters, you have to manually configure separate cluster-ids for each cluster. By default, the Yarn cluster and the Yarn session automatically generate cluster-ids based on Yarn application id. A manual configuration overrides this behaviour in Yarn. Specifying a cluster-id with the -z CLI option, in turn, overrides manual configuration.
- **Storage directory** (required): JobManager metadata is persisted in the file system *storageDir* and only a pointer to this state is stored in ZooKeeper.
@@ -102,7 +102,7 @@ After configuring the masters and the ZooKeeper quorum, you can use the provided
high-availability: zookeeper
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.path.root: /flink
-high-availability.zookeeper.path.namespace: /cluster_one # important: customize per cluster
+high-availability.zookeeper.path.cluster-id: /cluster_one # important: customize per cluster
high-availability.zookeeper.storageDir: hdfs:///flink/recovery</pre>
2. **Configure masters** in `conf/masters`:
@@ -187,7 +187,7 @@ high-availability: zookeeper
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
-high-availability.zookeeper.path.namespace: /cluster_one # important: customize per cluster
+high-availability.zookeeper.path.cluster-id: /cluster_one # important: customize per cluster
yarn.application-attempts: 10</pre>
3. **Configure ZooKeeper server** in `conf/zoo.cfg` (currently it's only possible to run a single ZooKeeper server per machine):
[2/3] flink git commit: [FLINK-7241] Remove cluster-id from YARN HA
setup template in doc
Posted by al...@apache.org.
[FLINK-7241] Remove cluster-id from YARN HA setup template in doc
When running a YARN cluster the cluster-id is automatically derived.
Acutally setting a cluster-id can lead to problems if you start multiple
per-job YARN sessions with the same cluster-id by accident.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d628e7a7
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d628e7a7
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d628e7a7
Branch: refs/heads/master
Commit: d628e7a7b45aa3d82a9195dc227c1067151d9c99
Parents: a671035
Author: Aljoscha Krettek <al...@gmail.com>
Authored: Fri Jul 21 17:46:25 2017 +0200
Committer: Aljoscha Krettek <al...@gmail.com>
Committed: Mon Jul 24 18:36:48 2017 +0200
----------------------------------------------------------------------
docs/setup/jobmanager_high_availability.md | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/d628e7a7/docs/setup/jobmanager_high_availability.md
----------------------------------------------------------------------
diff --git a/docs/setup/jobmanager_high_availability.md b/docs/setup/jobmanager_high_availability.md
index e770fa8..8754537 100644
--- a/docs/setup/jobmanager_high_availability.md
+++ b/docs/setup/jobmanager_high_availability.md
@@ -187,7 +187,6 @@ high-availability: zookeeper
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
-high-availability.zookeeper.path.cluster-id: /cluster_one # important: customize per cluster
yarn.application-attempts: 10</pre>
3. **Configure ZooKeeper server** in `conf/zoo.cfg` (currently it's only possible to run a single ZooKeeper server per machine):