You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2026/06/08 19:18:00 UTC
[jira] [Created] (NIFI-16006) Cluster coordinator can disconnect a freshly-rejoined node
Mark Payne created NIFI-16006:
---------------------------------
Summary: Cluster coordinator can disconnect a freshly-rejoined node
Key: NIFI-16006
URL: https://issues.apache.org/jira/browse/NIFI-16006
Project: Apache NiFi
Issue Type: Bug
Reporter: Mark Payne
When a NiFi cluster node is disconnected through the REST API (PUT /controller/cluster/nodes/\{id} with state DISCONNECTING) and a heartbeat from that node arrives at the coordinator immediately afterward — because the heartbeat was created and dispatched on the node before the node received the disconnection notification — AbstractHeartbeatMonitor logs:
Ignoring received heartbeat from disconnected node <host:port>. Node was disconnected due to [User Disconnected Node]. Issuing disconnection request.
and enqueues a new DISCONNECTION_REQUEST directed at that node. NodeClusterCoordinator's Disconnect <nodeId> thread retries the delivery indefinitely until the notification is successfully received, logging Failed to notify <host:port> that it has been disconnected on each failed attempt.
If the node's JVM is stopped before the retry succeeds and then restarted, the new JVM sends a Cluster Connection Request to the coordinator, is accepted as CONNECTING, and within milliseconds the queued DISCONNECTION_REQUEST (originally intended for the old JVM) is finally delivered to it. StandardFlowService on the new JVM processes the disconnection-notification and flips the node to "Not Clustered", but not before the node's first heartbeat reaches the coordinator. The coordinator therefore observes the node as CONNECTED for one heartbeat cycle and only catches up to reality when it times out the missing heartbeats (~17 seconds with the default 2-second heartbeat interval and 8x missing-heartbeat threshold).
Net effect: any observer that reads cluster state during the brief false-CONNECTED window — including REST clients, the UI, and system-test helpers such as NiFiSystemIT.waitForAllNodesConnected() — sees a healthy cluster and proceeds. The cluster silently degrades several seconds later with the node DISCONNECTED for "Lack of Heartbeat" and no auto-reconnect.
h3. Root cause:
Two cooperating issues, either of which alone would prevent the bug:
NodeClusterCoordinator does not cancel pending DISCONNECTION_REQUEST retry attempts when it receives a fresh Connection Request from the same node. The retry succeeds against the new JVM and is interpreted as a legitimate user-issued disconnect.
StandardFlowService does not track a connection-generation identifier that would let it discard a DISCONNECTION_REQUEST that is older than its current Connection Request. As a result, the freshly-joined node blindly processes the stale message and disconnects itself.
h3. Reproduction steps:
Start a 2-node NiFi cluster.
Confirm both nodes are CONNECTED.
PUT /nifi-api/controller/cluster/nodes/\{node2Id}
with status DISCONNECTING.
Immediately kill the node 2 JVM (do not wait for the disconnect retries on the coordinator to settle).
Restart the node 2 JVM.
Poll GET /nifi-api/controller/cluster/nodes from the coordinator: node 2 reports CONNECTED briefly, then transitions to DISCONNECTED with disconnect reason Lack of Heartbeat ~16-17 seconds later. Nothing reconnects it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)