[jira] [Work logged] (CAMEL-13338) ConsumerRebalanceListener is not registered when topicIsPattern is turned off. Cauing message loss or too many duplicates

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-13338?focusedWorklogId=216241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-216241 ]

ASF GitHub Bot logged work on CAMEL-13338:
------------------------------------------

                Author: ASF GitHub Bot
            Created on: 20/Mar/19 14:51
            Start Date: 20/Mar/19 14:51
    Worklog Time Spent: 10m 
      Work Description: viswaramamoorthy commented on pull request #2833: CAMEL-13338 - Fix to use Kafka client KafkaConsumer.subscribe API wit…
URL: https://github.com/apache/camel/pull/2833
 
 
   …h ConsumerRebalanceListener registration when topicIsPattern is set to false
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 216241)
            Time Spent: 10m
    Remaining Estimate: 0h

> ConsumerRebalanceListener is not registered when topicIsPattern is turned off. Cauing message loss or too many duplicates
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-13338
>                 URL: https://issues.apache.org/jira/browse/CAMEL-13338
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kafka
>    Affects Versions: 2.23.0
>            Reporter: Viswa Ramamoorthy
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Not registering ConsumerRebalanceListener when topicIsPattern turned off, causes either too many duplicate messages getting delivered to message processing layer or causes message loss depending on autoOffsetReset flag value.
>  
> Issue is noticed with below test scenario
> Topic: e.g. TestMessage
> Number of partitions: 25 (this is not completely relevant to issue)
> ConsumerUri: kafka:<kafka-broker-uris>?topic=TestMessage&groupId=TestMessage_GROUP&consumersCount=5&autoCommitEnable=false&offsetRepository=#<DB-state-repository>
> Number processes running consumers: 2 or more; e.g. two instances of Docker containers listening from same topic each with 5 consumer threads
>  
> Please note that offset management is done in DB with an implementation of Camel's StateRepository interface to eliminate too duplicate messages arriving at the message processor (at most one message processing needed for my use case)
>  
> When a second process instance (say second container) brought up, since ConsumerRebalanceListener is not registered, partitions start processing at earliest offsets if autoOffsetReset is set to earliest as expected. In this case, too many duplicate messages arrive at the message processing layer. If autoOffsetReset is set to latest, message loss occurs for certain partitions. If autoOffsetReset is set to none, no valid offset exception thrown by broker
>  
> Issue would be noticed consistently when simulating slow message processing for the above test scenario
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)