[jira] [Resolved] (CAMEL-24457) Camel Zookeeper Cluster Split brain issue when leader is isolated

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-24457.
---------------------------------
    Resolution: Fixed

> Camel Zookeeper Cluster Split brain issue when leader is isolated
> -----------------------------------------------------------------
>
>                 Key: CAMEL-24457
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24457
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-zookeeper, camel-zookeeper-master
>    Affects Versions: 4.18.3
>         Environment: 3 Linux RHEL machines 
> All 3 has ZK on it
> And 2 Has the the camel application in cluster mode Active/Passive 
>            Reporter: mustafa kamal
>            Priority: Major
>              Labels: bug, split-brain
>             Fix For: 4.23.0
>
>
> ------------------------------------------------------------------------------------------------------------------
> Problem Description:
> ------------------------------------------------------------------------------------------------------------------
> When the ZooKeeper cluster becomes unavailable, the Camel route managed by "ClusteredRoutePolicy" never stops, even though the node has lost its ZooKeeper session. Beyond the route not stopping, this also introduces a split-brain scenario: if one node becomes isolated from ZooKeeper while others remain connected, a new leader election runs on the healthy side and a second node wins leadership and starts the same route — now two nodes are running the same route simultaneously with no coordination, which can cause data corruption, duplicate processing, or conflicting writes depending on what the route does.
>  
> ------------------------------------------------------------------------------------------------------------------
> Root Cause:
> ------------------------------------------------------------------------------------------------------------------
> The root cause is a timing issue between two threads in "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()". When ZooKeeper goes down, Curator's "ConnectionStateManager" thread fires a "SUSPENDED"/"LOST" state change, which causes "LeaderSelectorListenerAdapter" to interrupt the thread blocked inside "takeLeadership()". The "BlockingTask" handles the interrupt correctly and exits its loop. Execution then reaches the leadership-lost event at:
>  
> fireLeadershipChangedEvent(getLeader().orElse(null)); // ZooKeeperClusterView.java line 155
>  
> This fires "ClusteredRoutePolicy.leadershipChanged()", which, regardless of the argument passed, always calls back into:
> setLeader(clusterView.getLocalMember().isLeader());  // ClusteredRoutePolicy.java line 376
>  
> Which resolves to: leaderSelector.hasLeadership()  // CuratorLocalMember.isLeader(), line 162
> This returns "true" at this point because Curator's guarantees that "hasLeadership()" only becomes "false" after "takeLeadership()" returns to the "LeaderSelector" internals. The event fires from inside "takeLeadership()", so the answer is always "true", "ClusteredRoutePolicy" sees no leadership change, and the route keeps running indefinitely.
>  
> ------------------------------------------------------------------------------------------------------------------
> Consequences:
> ------------------------------------------------------------------------------------------------------------------
> The consequence is:
> - The [isolated node] keeps running the route because "hasLeadership()" is still "true" at the moment the event fires
> - The [healthy side] elects a new leader, which also starts the same route
> - Both nodes now process the same workload simultaneously with no mutual exclusion — a classic [split-brain]
>  
>  
> ------------------------------------------------------------------------------------------------------------------
> Solution:
> ------------------------------------------------------------------------------------------------------------------
> Introduce a "volatile boolean leader" flag inside "ZooKeeperClusterView" that is owned and controlled by the view itself,  rather than delegating to "leaderSelector.hasLeadership()".
> Change "CuratorLocalMember.isLeader()" to return this flag instead.
> In "takeLeadership()", set the flag to "true" before firing the leadership-gained event, and in the "finally" block set it to "false" before firing the leadership-lost event. This guarantees that when "ClusteredRoutePolicy" calls back into "isLeader()" during the event,  it reads "false", which causes "stopManagedRoutes()" to be called and the route to stop correctly  before any other node can win the election and start it.
> Additionally, "leaderSelector.autoRequeue()" should be called in "doStart()" so that after losing leadership due to a ZooKeeper disconnect, the node automatically re-enters the election when ZooKeeper reconnects and the route can start again on whichever node wins.
>  
> - "ZooKeeperClusterView": add "volatile boolean leader" field
> - "ZooKeeperClusterView.CuratorLocalMember.isLeader()": return "leader" flag instead of "leaderSelector.hasLeadership()"
> - "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()": set "leader = true" on entry, wrap task in "try/finally", set "leader = false" before firing the lost event
> - "ZooKeeperClusterView.doStart()": add "leaderSelector.autoRequeue()" to re-enter election after reconnect
>  
> ------------------------------------------------------------------------------------------------------------------
> How to reproduce:
> ------------------------------------------------------------------------------------------------------------------
> 1. Start two instances of a Camel application using "ZooKeeperClusterService" with "ClusteredRouteController"
> 2. Confirm one instance is leader and its route is running
> 3. Isolate the leader node from all ZooKeeper nodes (e.g. firewall rules or kill ZK nodes)
> 4. Observe on the healthy side: a new leader is elected and its route starts
> 5. Observe on the isolated node: the route never stops — both nodes are now running the same route simultaneously (split-brain)
> 6. Expected: the isolated node's route stops as soon as its ZooKeeper session is lost



--
This message was sent by Atlassian Jira
(v8.20.10#820010)