[jira] [Updated] (CAMEL-25843) camel-hazelcast - a reliable topic consumer that falls behind the ringbuffer is removed by Hazelcast and receives nothing again

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25843:
--------------------------------
    Fix Version/s: 4.24.0

> camel-hazelcast - a reliable topic consumer that falls behind the ringbuffer is removed by Hazelcast and receives nothing again
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25843
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25843
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-hazelcast
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> With {{reliable=true}} the hazelcast-topic consumer ({{HazelcastTopicConsumer}}, main 20cdb309f7d0) listens to a reliable topic with a plain {{MessageListener}} ({{CamelMessageListener}}). A reliable topic keeps its messages in a ringbuffer of limited capacity (10000 by default, optionally with a time to live). When a listener falls behind by more than that, the messages it has not read yet are overwritten. Hazelcast then removes a listener that is not loss tolerant, for good: {{MessageRunner.accept}} (Hazelcast 5.4.0) calls {{cancel()}} when the read result shows lost messages and {{isLossTolerable}} is false, and a plain {{MessageListener}} is wrapped in {{ReliableMessageListenerAdapter}}, whose {{isLossTolerant()}} returns false.
> So one burst that the route cannot keep up with (the route runs on Hazelcast's listener thread; a slow processor, a downstream system that slows down) ends the consumer: the route stays started, receives nothing again until it is restarted, and Camel logs nothing (Hazelcast logs one warning through its own logging). The overwritten messages are lost either way; the issue is that every message after them is lost too.
> *Fix*: the reliable topic consumer registers {{CamelReliableMessageListener}}, a {{ReliableMessageListener}} that is loss tolerant: it logs a WARN with the number of messages lost ("The consumer of the reliable topic news lost 21 messages: they were overwritten in the ringbuffer before they were received") and goes on with the oldest message still in the ringbuffer. It starts with the messages published from then on ({{retrieveInitialSequence}} -1), as the plain listener did. The plain topic is unchanged.
> *Test*: new {{HazelcastReliableTopicConsumerLossTest}} (an embedded member, ringbuffer capacity 10): the route is busy with a first message while 30 more are published, then one more message is published. On main it is never received: {{mock://news Received message count. Expected: <1> but was: <0>}}. With the fix it is, and the WARN reports 21 lost messages. camel-hazelcast unit tests with the fix: 243 run, 0 failures, 1 error ({{HazelcastUserConfigSerializationFilterWarningTest.warnsForUserClientConfigWithoutFilter}}, a client that cannot reach a cluster in the test environment; it errors the same way without the change).
> *How it was found*: variant analysis of CAMEL-25815 (pattern: the server side ends a consumer's subscription and the consumer never subscribes again), then the JUnit test.
> *Affected*: main, camel-4.18.x and camel-4.14.x (the same plain listener, Hazelcast 5.4.0; read, not run).
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)