[jira] [Resolved] (CAMEL-25315) camel-zookeeper-master - a master whose consumer failed to start never consumes, and a consumer started while the leadership is lost or the route stops keeps running

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-25315.
---------------------------------
    Resolution: Fixed

Merged in https://github.com/apache/camel/pull/27391 (commit e670025a49a4) for 4.23.0.

_Claude Code on behalf of davsclaus_

> camel-zookeeper-master - a master whose consumer failed to start never consumes, and a consumer started while the leadership is lost or the route stops keeps running
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25315
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25315
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-zookeeper-master
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.23.0
>
>
> {{MasterConsumer}} starts the delegate consumer from the {{CHANGED}} group event (the group's own thread, {{onLockOwned}}) and stops it from {{DISCONNECTED}} (called directly on the Curator connection-state thread, and by {{ZooKeeperGroup.close()}} on the closing thread) and from {{doStop}}. Nothing synchronises them, and {{onLockOwned}} only acts {{if (delegate == null)}}:
> {code:java}
> if (delegate == null) {
>     try {
>         ServiceHelper.startService(endpoint.getConsumerEndpoint());
>         delegate = endpoint.getConsumerEndpoint().createConsumer(processor);
>         ...
>         groupListener.updateState(thisNodeState);
>         ServiceHelper.startService(delegate);
>     } catch (Exception e) {
>         LOG.error("Failed to start master consumer for: {}", endpoint, e);
>     }
> }
> {code}
> * *A failed start is never retried.* When the delegate cannot start at the moment the node is elected (JMS broker, FTP or database server not reachable yet), the failed consumer stays in {{delegate}}, so every later leadership event skips the start. The node keeps the ZooKeeper leadership, the other nodes stay standby, and nobody consumes until this node loses its connection or the route is restarted. The only trace is one ERROR log line.
> * *A consumer started after a disconnect keeps running on a node that is not the master.* If the connection is lost ({{SUSPENDED}}/{{LOST}} -> {{DISCONNECTED}} -> {{stopConsumer()}}) between the null check and the assignment of the new delegate (endpoint start, {{createConsumer}}), the stop finds nothing, and the delegate is then assigned and started. Another node becomes the master: two nodes consume. When this node reconnects as a standby nothing stops it (standby events do nothing).
> * *The same after a route stop:* {{doStop}} stops nothing, and {{close()}} only calls {{DISCONNECTED}} again after waiting up to 5 seconds for the group thread, so a start that takes longer leaves a consumer running after the route (or the CamelContext) stopped.
> camel-master's {{MasterConsumer}} handles these cases (leadership lock, start attempts with a backoff, a start that finishes after the leadership was lost is stopped).
> h3. Reproduction
> {{MasterConsumerLeadershipTest}} (camel-zookeeper-master, no ZooKeeper: a {{ManagedGroupFactory}} in the registry returns a fake group whose events the test fires; the delegate endpoint can fail a start or hold {{createConsumer}} on a latch), on main:
> {noformat}
> testFailedStartIsRetried: the first start fails
>   The master must start its consumer after a failed start ==> expected: <1> but was: <0> within 20 seconds.
> testDisconnectedWhileCreatingConsumer: DISCONNECTED while the group thread is in createConsumer
>   A node that lost the leadership must not consume ==> expected: <0> but was: <1>
> testStoppedWhileCreatingConsumer: stopRoute while the group thread is in createConsumer
>   A stopped route must not consume ==> expected: <0> but was: <1>
> {noformat}
> No sleeps: latches, Awaitility.
> The defect was found with a TLA+ model of the consumer (group thread, Curator connection thread, stopping thread; the null check, the creation, the start and the stop as separate steps; {{close()}} waiting at most 5 s): "a master that received a leadership event consumes" is violated in 3 steps with one failed start, and "no delegate runs on a stopped route or a node that is not the master" in 5 steps (event, connection lost, stopConsumer, create, start) and in 7 steps for the route stop. A first version of the model let the start read a stale delegate; the Java reads the field again ({{startService(delegate)}}), which the corrected model follows. With the fix all properties hold, with up to 3 events, 2 failed starts, a disconnect and a stop, and TLC's deadlock check on; mutations of the fix without the generation check (a disconnect during the start leaves a zombie) or without the retry violate them. The model does not cover reconnecting as master after a disconnect.
> h3. Proposed fix
> A {{leadershipLock}} and a generation counter (incremented by {{onDisconnected}} and {{doStop}}): a start is marked under the lock after checking again that the group is connected and this node is the master ({{ZooKeeperGroup}} clears {{connected}} before it calls {{DISCONNECTED}}); the consumer is created and started without holding the lock (it can take long); it is published under the lock only if the generation did not change, otherwise it is stopped. A failed start is retried after 5 seconds by a scheduled task of the same generation (cancelled by a disconnect or stop), and the started state is published to the group once per leadership term, so a retry does not trigger another group event (a new state with a random container id would). This follows camel-master's {{MasterConsumer}} (CAMEL-24583, CAMEL-24626) in its locking and lost-leadership checks, with two differences: camel-zookeeper-master has no {{backOffDelay}}/{{backOffMaxAttempts}} options, so the delay is fixed at camel-master's default of 5 seconds and the retries go on while this node is the master (camel-master gives up after 10 attempts and stays leader without consuming); and each attempt creates a new delegate consumer (a failed {{start()}} already stopped it), where camel-master reuses one. Each failed attempt is logged at WARN with its stack trace.
> With the fix the new tests and the camel-zookeeper-master unit tests (14) pass; the integration tests need Docker and were not run.
> Affected: 4.14.x, 4.18.x and main (same code, GitHub contents API); {{MasterConsumer}} unchanged since 2021.
> Duplicate check (2026-10-04): JIRA component camel-zookeeper-master and text "zookeeper-master"/"zookeepermaster" (53 issues; CAMEL-24457 and CAMEL-24545 are about the camel-zookeeper cluster service, CAMEL-17226 namespaces; none about the master consumer's start or races). No open PR touches the component.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)