You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2019/08/26 15:17:00 UTC

[jira] [Created] (NIFI-6589) Leader Election should cache results obtained from ZooKeeper

Mark Payne created NIFI-6589:
--------------------------------

             Summary: Leader Election should cache results obtained from ZooKeeper
                 Key: NIFI-6589
                 URL: https://issues.apache.org/jira/browse/NIFI-6589
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
            Reporter: Mark Payne
            Assignee: Mark Payne


In order to determine which node in a cluster is the Cluster Coordinator, a node must make a request to ZooKeeper. That means that if we have N nodes in a cluster, then we must ask ZooKeeper for each request at least (N+1) times (and no more than N+2) who is the Cluster Coordinator. This is done because when the request comes in, the node must determine whether or not it is the Cluster Coordinator. If so, it must replicate the request to each node. If not, it must forward the request to the Cluster Coordinator, which will then do so. When the request is replicated, it will again check if it is the cluster coordinator. If we instead cache the result of querying ZooKeeper for a short period of time, say 1 minute, we can dramatically decrease the number of times that we hit ZooKeeper. If the Coordinator / Primary Node changes in the mean time, it should still be notified of the change asynchronously.

The polling is done currently because we've seen situations where the asynchronous notification did not happen. But if we update the code so that we cache the results, this means that we will also update the code for caching results of which node is Primary Node. This is a benefit as well, because currently we don't poll for this and as a result, if we do happen to miss the notification, we could theoretically have 2 nodes running processors should only run on Primary Node.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)