[jira] [Commented] (NIFI-16006) Cluster coordinator can disconnect a freshly-rejoined node

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088589#comment-18088589 ] 

ASF subversion and git services commented on NIFI-16006:
--------------------------------------------------------

Commit 42c3006b12304d309ee573c3069c9f1ff4806920 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=42c3006b123 ]

NIFI-16006: Addressed issue where a node can be disconnected, then so… (#11331)

* NIFI-16006: Addressed issue where a node can be disconnected, then soon after re-connect but then be quickly told to disconnect due to a queued up 'Disconnect' message from the original disconnection. Now, we use a 'generation' flag so we know to ignore the message, and we also cancel the background task that is trying to deliver it.

> Cluster coordinator can disconnect a freshly-rejoined node
> ----------------------------------------------------------
>
>                 Key: NIFI-16006
>                 URL: https://issues.apache.org/jira/browse/NIFI-16006
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When a NiFi cluster node is disconnected through the REST API (PUT /controller/cluster/nodes/\{id} with state DISCONNECTING) and a heartbeat from that node arrives at the coordinator immediately afterward — because the heartbeat was created and dispatched on the node before the node received the disconnection notification — AbstractHeartbeatMonitor logs:
> Ignoring received heartbeat from disconnected node <host:port>. Node was disconnected due to [User Disconnected Node]. Issuing disconnection request.
> and enqueues a new DISCONNECTION_REQUEST directed at that node. NodeClusterCoordinator's Disconnect <nodeId> thread retries the delivery indefinitely until the notification is successfully received, logging Failed to notify <host:port> that it has been disconnected on each failed attempt.
> If the node's JVM is stopped before the retry succeeds and then restarted, the new JVM sends a Cluster Connection Request to the coordinator, is accepted as CONNECTING, and within milliseconds the queued DISCONNECTION_REQUEST (originally intended for the old JVM) is finally delivered to it. StandardFlowService on the new JVM processes the disconnection-notification and flips the node to "Not Clustered", but not before the node's first heartbeat reaches the coordinator. The coordinator therefore observes the node as CONNECTED for one heartbeat cycle and only catches up to reality when it times out the missing heartbeats (~17 seconds with the default 2-second heartbeat interval and 8x missing-heartbeat threshold).
> Net effect: any observer that reads cluster state during the brief false-CONNECTED window — including REST clients, the UI, and system-test helpers such as NiFiSystemIT.waitForAllNodesConnected() — sees a healthy cluster and proceeds. The cluster silently degrades several seconds later with the node DISCONNECTED for "Lack of Heartbeat" and no auto-reconnect.
> h3. Root cause:
> Two cooperating issues, either of which alone would prevent the bug:
> NodeClusterCoordinator does not cancel pending DISCONNECTION_REQUEST retry attempts when it receives a fresh Connection Request from the same node. The retry succeeds against the new JVM and is interpreted as a legitimate user-issued disconnect.
> StandardFlowService does not track a connection-generation identifier that would let it discard a DISCONNECTION_REQUEST that is older than its current Connection Request. As a result, the freshly-joined node blindly processes the stale message and disconnects itself.
> h3. Reproduction steps:
> Start a 2-node NiFi cluster.
> Confirm both nodes are CONNECTED.
> PUT /nifi-api/controller/cluster/nodes/\{node2Id}
> with status DISCONNECTING.
> Immediately kill the node 2 JVM (do not wait for the disconnect retries on the coordinator to settle).
> Restart the node 2 JVM.
> Poll GET /nifi-api/controller/cluster/nodes from the coordinator: node 2 reports CONNECTED briefly, then transitions to DISCONNECTED with disconnect reason Lack of Heartbeat ~16-17 seconds later. Nothing reconnects it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)