You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Claus Ibsen (Jira)" <ji...@apache.org> on 2026/10/14 11:49:00 UTC
[jira] [Resolved] (CAMEL-25613) camel-jmx - consumer with reconnectOnConnectionFailure=true closes and replaces a working connection every reconnectDelay for ever after a lost connection
[ https://issues.apache.org/jira/browse/CAMEL-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-25613.
---------------------------------
Resolution: Fixed
> camel-jmx - consumer with reconnectOnConnectionFailure=true closes and replaces a working connection every reconnectDelay for ever after a lost connection
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25613
> URL: https://issues.apache.org/jira/browse/CAMEL-25613
> Project: Camel
> Issue Type: Bug
> Components: camel-jmx
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> All of this is about a remote JMX consumer with {{reconnectOnConnectionFailure=true}}. The option is off by default; it is the option one sets so that the consumer survives an outage of the monitored JVM, and the defect follows every such outage. {{JMXConsumer}}, lines at main a7169d7c6ab6:
> * {{initNetworkConnection}} (139-154) closes the previous connector (142), connects (150), adds the connection listener (151) and stores the connection id (152);
> * {{ConnectionNotificationListener.handleNotification}} (190-207) ignores a notification whose connection id is not the stored one (193) and calls {{scheduleReconnect()}} for {{NOTIFS_LOST}}, {{CLOSED}} and {{FAILED}} (196-201);
> * {{scheduleReconnect}} (213-226) schedules one task per call; the task calls {{initNetworkConnection()}} and {{addNotificationListener()}}, and any exception schedules it again (220).
> h3. 1. One lost connection schedules two reconnects, and the replacement never stops
> The RMI connector of the JDK reports a connection that can no longer be used with {{jmx.remote.connection.failed}} and then, from the close that follows, with {{jmx.remote.connection.closed}}, both with the id of that connection (observed with JDK 21; {{RMIConnector.RMIClientCommunicatorAdmin.gotIOException}} sends the first and calls {{close(true)}}, which sends the second). Both pass the id check (193), so two reconnect tasks are scheduled.
> When the server is back, the first task connects and subscribes. The second task then closes that working connector (142). Closing a connector whose connection works sends {{jmx.remote.connection.closed}} with the id of that connection to the connection listener, synchronously, inside {{close()}}; the id is still the stored one (it is replaced at 152), so the listener schedules a third task; and so on. Result: after an outage the consumer closes and replaces a working connection every {{reconnectDelay}} (10 seconds by default) for as long as the route runs, with a WARN "Lost JMX connection" each time, and notifications that the MBean sends between a close and the next subscription are lost.
> Observed in a local run with main's {{JMXConsumer}}, a JMX server in a second JVM that was killed and started again on the same port, and {{reconnectDelay=1}}: the connector delivered {{failed}} and {{closed}} with the stored id from one thread, the consumer logged "Lost JMX connection" and "Delaying JMX consumer reconnection" twice, and the new server accepted 16 connections from the consumer in the first 8 seconds (two every second).
> The "closed" notification of its own connector keeps the replacement going in two more cases:
> * after one {{NOTIFS_LOST}} (the server dropped notifications for this client; the connection itself is fine);
> * when an attempt connected but {{addNotificationListener}} failed, typically with {{InstanceNotFoundException}}: the JMX connector of a restarting application is usually reachable before the application has registered its MBeans.
> h3. 2. The consumer does not follow the connection id after the connector reconnected by itself
> When the server object is gone but the server still answers ({{NoSuchObjectException}}, for example a connector server that was stopped and started again inside a JVM that keeps running), the RMI connector tries once to re-establish its connection. If that works it sends {{jmx.remote.connection.opened}} with a new connection id, and it adds again only the listeners whose MBean exists at that moment (read in the JDK sources: {{RMIClientCommunicatorAdmin.doStart}} and {{reconnectNotificationListeners}}). The consumer ignores that notification and keeps the old id, so:
> * every later {{FAILED}} / {{CLOSED}} / {{NOTIFS_LOST}} carries the new id and is ignored (193): the consumer never reconnects again, although {{reconnectOnConnectionFailure=true}}, and nothing is logged;
> * when the MBean was not registered at the moment of the reconnection, the consumer has a connection and no subscription, receives nothing, and logs nothing.
> h3. 3. A notification after stop reconnects the stopped consumer
> The lost-connection branch of the listener (196-201) does not check whether the consumer still runs. {{doStop}} shuts the scheduled executor down and forgets it (266-269); a notification that is handled after that makes {{getExecutor()}} (232-238) create a new executor, and the stopped consumer connects again.
> h3. Reproduction
> New {{JMXConsumerReconnectTest}} (RMI registry and connector server in the test JVM, set up like the existing {{JMXRobustRemoteConnectionTest}}; {{reconnectDelay=1}}; the connector server counts the client connections it accepts). On main all five tests fail:
> * {{FAILED}} and then {{CLOSED}} with the current connection id are given to the connection listener: one new connection is expected and then none for three seconds. On main: {{the consumer connected again although it was subscribed on a working connection ==> expected: <false> but was: <true>}};
> * a {{NOTIFS_LOST}} notification for the current connection: one new connection and then none. On main the same message;
> * the server is stopped and started again without the MBean, the test waits for two new connections, registers the MBean and waits for a notification; after that the connections must stop. On main: {{the consumer kept connecting although it was subscribed on a working connection}} (when the connector reported the loss) or {{the consumer did not keep trying to subscribe after the server came back without the MBean}} (when the connector reconnected by itself; which of the two happens depends on timing);
> * an {{OPENED}} notification with another connection id: the consumer must open a connection of its own again. On main: {{the consumer did not subscribe again after the connector had reconnected by itself ==> expected: <true> but was: <false>}};
> * the route is stopped and {{FAILED}} with the last connection id is given to the listener: no connection may follow. On main: {{the consumer connected again although it was stopped ==> expected: <false> but was: <true>}}.
> h3. Proposed fix
> * {{scheduleReconnect}} schedules at most one reconnect at a time: a flag is set with {{compareAndSet}} when a reconnect is scheduled, cleared when the task begins (so that a failure reported during the attempt schedules the next one) and reset in {{doStop}}. It schedules nothing when the consumer is not allowed to run, and the task ends at once in that case.
> * {{initNetworkConnection}} removes the connection listener from the previous connector before it closes it.
> * An {{OPENED}} notification with another id: the consumer stores the new id and, with {{reconnectOnConnectionFailure=true}}, schedules its normal reconnect, which subscribes again on a new connection (it cannot know whether the connector kept its listener). A WARN line is logged; with {{reconnectOnConnectionFailure=false}} only a WARN line that the consumer may not be subscribed anymore.
> * The listener ignores a lost connection when the consumer is not allowed to run.
> * The fields that the notification threads read are volatile.
> One consequence to know: when the connector re-established the connection by itself and all listeners were added again, main keeps receiving without a gap (and is deaf to later failures). With the fix the consumer replaces that connection once, {{reconnectDelay}} later; notifications that the MBean sends between the close and the new subscription are lost that one time.
> Not changed: with {{testConnectionOnStartup=false}} a delayed start that connects but cannot add the listener still logs "An unrecoverable exception has occurred" at ERROR and does not try again; {{doStop}} still fails when the server cannot be reached; a reconnect attempt that is already connecting when the consumer is stopped is not interrupted and may leave a connection open; the WARN "Lost JMX connection" is still logged for each of the two notifications.
> camel-jmx tests with the fix: 58 run, 0 failures, 0 skipped. With the fix and the same second-JVM run as above the new server accepted one connection and the next notification of the MBean was delivered. Not run: a JMX server on another host.
> How it was found: a sweep of the reconnect code of the components, then a small TLA+ model (server up or down, MBean registered, connector state, stored id, pending and running reconnect, the "failed" and "closed" pair, stop). For main TLC reports two pending reconnects after one lost connection (4 steps), a replaced connection without a cause, "nothing pending, server up, MBean registered, not subscribed", a run that never ends subscribed with nothing pending, and a reconnect scheduled after stop. With the fix the properties hold for three environment events and one stop (739 states). The model is a sketch; the tests are the evidence.
> Affected: main (a7169d7c6ab6; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x have the same id check, the same {{initNetworkConnection}} and the same {{scheduleReconnect}} (read, not run).
> Duplicate check (2026-10-13): JIRA text "jmx reconnectOnConnectionFailure" (CAMEL-5613), "JMXConsumer reconnect" (CAMEL-19527, tests only), "jmx consumer connection notification" (nothing related); GitHub pull requests "JMXConsumer", "camel-jmx reconnect": #2052 (2017, close the connector on shutdown), none for this.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)