You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/08/14 17:31:46 UTC
[jira] [Commented] (FLINK-2291) Use ZooKeeper to elect JobManager
leader and send information to TaskManagers
[ https://issues.apache.org/jira/browse/FLINK-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697230#comment-14697230 ]
ASF GitHub Bot commented on FLINK-2291:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/1016
[FLINK-2291] [runtime] Adds high availability support via ZooKeeper
## Idea
This PR introduces cluster high availability via ZooKeeper. The idea is to use ZooKeeper to do leader election among a group of registered `JobManagers`. The elected leader writes his akka connection URL and his assigned leader session ID to ZooKeeper from where the `TaskManagers` can retrieve it.
## Activation
In order to use the high availability mode, one has to select Flinks *zookeeper* **recovery mode** and specify a valid **ZK quorum**. Both is done in the `flink-conf.yaml` by setting `recovery.mode: zookeeper` and `ha.zookeeper.quorum: address1:2181[,...],addressX:2181` where the zk quorum addresses point to ZooKeeper servers.
## Implementation
In order to support HA ZK and also the standalone recovery mode (no HA), this PR introduces the `LeaderElectionService` and the `LeaderRetrievalService`. The former service is used by leader contenders to be elected as the leader. The latter is used to obtain the address of the current leader. In standalone mode (`StandaloneLeaderElectionService`, `StandaloneLeaderRetrievalService`), these services just return the `JobManager` address which was found in the Flink configuration. With ZooKeeper, the services use the Curator framework to connect to the ZooKeeper quorum to do leader election and to read the ZkNode which contains the information of the current leader.
In the wake of introducing these services, the `FlinkMiniCluster` was also adapted to support HA with ZooKeeper. The `ForkableFlinkMiniCluster` starts automatically a ZK TestingCluster if `recovery.mode` was set to **zookeeper** in the provided configuration and if the `ha.zookeeper.quorum` was not set.
## Limitations
Currently in HA ZK mode, one web server is started per `JobManager`. The reason for not having a dedicated web server is that parts of the information it requires are not serializable yet. Thus, each `JobManager` has a local web server showing its own state. Since the web servers are running dedicatedly for one `JobManager`, they don't know about the current leader session ID. Therefore, it is not possible to cancel jobs via the web servers. This is because a `CancelJob` message is a `RequiresLeaderSessionMessage`.
This PR depends on the PR #1015.
Since this PR touches a lot of files, a close review, as far as possible, would be helpful. I'll add some more descriptions of the internal workings after the weekend.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink ha
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1016.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1016
----
commit c29a6e67ea6b637405a6392e4c54702907679e95
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-14T13:06:06Z
[FLINK-2521] [tests] Adds TestLogger class which automatically logs the currently executed test names and the reasons for a failure.
Makes test bases extend TestLogger to add automatic test name logging
commit e582b3b6cfef9c2c2f6b038df67270ad4b0259d6
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-06-26T10:07:39Z
[FLINK-2291] [runtime] Add ZooKeeper support to elect a leader from a set of JobManager. The leader will then be retrieved from ZooKeeper by the TaskManagers.
Refactors FlinkMiniCluster to support multiple JobManager
Adds proper remote address resolution for actors
Clean up of LeaderElection and LeaderRetrievalService. Removes synchronization to avoid deadlock.
Adds ZooKeeper start option to TestBaseUtils.startCluster
Removes registration session IDs, using the leader session IDs instead. Sets the leader session ID
directly in the grantLeadership method. Let the LeaderElectionService select the leader session I
D. Return leader session ID to LeaderRetrievalListeners.
Removes direct ActorRef interaction
Introduces LeaderRetrievalService for the Client and the CliFrontend.
Make ApplicationClient to use the LeaderRetrievalService for JobManager resolution
Adds LeaderElection/Retrieval tests
Added test for exception forwarding from the CuratorFramework to a Contender
Adds test job submission with changing leaders
Adds new test cases for job cleanup after leader election change
Adds new LeaderChangeStateCleanup test case
Adds LeaderElectionRetrievalTestingCluster
commit 0e7b8b777a051159e621cb38f9bd1f6b0a84c778
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-07-29T14:52:38Z
Introduces ListeningBehaviour for job submissions
commit aae84f607f1fd2609b620c1baef21f022f1c099c
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-07-31T13:49:58Z
Relocation of org.apache.curator in flink-shaded-hadoop jar
Adds Apache ZooKeeper and Apache Curator to LICENSE and NOTICE files
Increases zookeeper connection timeout to 20000 ms for the KafkaITCase to fix failing tests on Travis
Increased timeouts of ZooKeeperLeaderElectionTest for Travis
commit 1b83c70495fe4d121815a162443dff461c8d79c3
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-04T14:25:17Z
Makes the WebInfoServer and the WebRuntimeMonitor to use the LeaderRetrievalService to retrieve the current leading JobManager
Adds proper synchronization to ZooKeeperLeaderElectionService. Fixes StateCheckpointedITCase and PartitionedStateCheckpointingITCase
commit 1a87994ab30e001bc6ba0561ca81144a4d049624
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-06T15:49:18Z
Adds configuration description for new ZooKeeper configuration values
Fixed port selection of JobManager at startup
Improves logging output
commit 291247bd65bebbf4dd0a0055d2355a766bd7c91c
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-07T15:56:17Z
Extends masters file to also specify the webui ports
commit 9f1d874df897f8a2bf36d7d45f05d79df99a8586
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-10T14:36:07Z
Adds proper network interface resolution by retrieving the current leader address
commit ca6522f376ea54881447c7efac147e382e78e4b2
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-12T12:45:19Z
Makes the ZooKeeperLeaderElectionService write the leader information in ephemeral nodes so that the information is deleted once the leader has terminated. Fixes a bug in the TaskManager due to call by name semantics of scheduler.scheduleOnce.
Adds jobManagerURL to TriggerTaskManagerRegistration message
commit ff40f2186bd85ca1ead0b07d77a13f5fcbebd821
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-13T12:48:29Z
Enables findConnectingAddress to use the ZooKeeperLeaderRetrievalService. This allows to test the connection to a possibly changing master node.
commit 8af6005f1abd2fbdbb397013681d6de5e94f1e58
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-13T14:05:35Z
Changes startup scripts to respect the recovery mode instead of the ZK_QUORUM
Adjust travis log file to only log zookeeper errors
commit f9da12a26779915bc0c046cd9f8444710fdbd6c4
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-14T10:30:25Z
Updates high availability setup guide
commit 8fe71ba7f41a0c0fbcf68339baaa44c4401aa0ee
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-08-14T14:11:05Z
Adds TestLogger to leader election tests
----
> Use ZooKeeper to elect JobManager leader and send information to TaskManagers
> -----------------------------------------------------------------------------
>
> Key: FLINK-2291
> URL: https://issues.apache.org/jira/browse/FLINK-2291
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager, TaskManager
> Affects Versions: 0.10
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Fix For: 0.10
>
>
> Use ZooKeeper to determine the leader of a set of {{JobManagers}} which will act as the responsible {{JobManager}} for all {{TaskManager}}. The {{TaskManager}} will get the address of the leader from ZooKeeper.
> Related Wiki: [https://cwiki.apache.org/confluence/display/FLINK/JobManager+High+Availability]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)