You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2019/08/26 15:17:00 UTC
[jira] [Created] (NIFI-6589) Leader Election should cache results
obtained from ZooKeeper
Mark Payne created NIFI-6589:
--------------------------------
Summary: Leader Election should cache results obtained from ZooKeeper
Key: NIFI-6589
URL: https://issues.apache.org/jira/browse/NIFI-6589
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
In order to determine which node in a cluster is the Cluster Coordinator, a node must make a request to ZooKeeper. That means that if we have N nodes in a cluster, then we must ask ZooKeeper for each request at least (N+1) times (and no more than N+2) who is the Cluster Coordinator. This is done because when the request comes in, the node must determine whether or not it is the Cluster Coordinator. If so, it must replicate the request to each node. If not, it must forward the request to the Cluster Coordinator, which will then do so. When the request is replicated, it will again check if it is the cluster coordinator. If we instead cache the result of querying ZooKeeper for a short period of time, say 1 minute, we can dramatically decrease the number of times that we hit ZooKeeper. If the Coordinator / Primary Node changes in the mean time, it should still be notified of the change asynchronously.
The polling is done currently because we've seen situations where the asynchronous notification did not happen. But if we update the code so that we cache the results, this means that we will also update the code for caching results of which node is Primary Node. This is a benefit as well, because currently we don't poll for this and as a result, if we do happen to miss the notification, we could theoretically have 2 nodes running processors should only run on Primary Node.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)