[jira] [Updated] (CAMEL-20044) camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed

["Mike Barlotta (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Barlotta updated CAMEL-20044:
----------------------------------
    Description: 
{*}Reproducing (intermittent){*}:
 * Configure camel kafka consumer with following:
 ** autoCommitEnable = false
 ** allowManualCommit = true
 ** autoOffsetReset = earliest
 ** maxPollRecords = 1
 ** breakOnFirstError = true
 * Produce a series of records to kafka record to both partitions.
 * Throw an exception that is unhandled
 * commit the offset in the onException block

*Expected behavior:*
 * Application should consume the record 1 more time, then move on to the next offset in the partition

*Actual behavior:*
 * Application will often work. Occasionally will use the offset from another partition and assign that to the partition where the record failed. This can then result in the consumer replaying messages instead of moving forward.

I put together a sample that can recreate the error. However, given its intermittent nature it may not fail on each run. I have included the logs from 3 different runs on my laptop from this test. Two of them show the error occurring. One of the them has a successful run. I have also provided more details in the README. 
 * [https://github.com/CodeSmell/CamelKafkaOffset]

This seems related to other issues with how Camel processes the _breakOnFirstError_ attribute. 
 * CAMEL-17925
 * CAMEL-14935
 * CAMEL-18350
 * CAMEL-19894

  was:
{*}Reproducing (intermittent){*}:
 * Configure camel kafka consumer with following:
 ** autoCommitEnable = false
 ** allowManualCommit = true
 ** autoOffsetReset = earliest
 ** maxPollRecords = 1
 ** breakOnFirstError = true
 * Produce a series of records to kafka record to both partitions.
 * Throw an exception that is unhandled
 * commit the offset in the onException block

*Expected behavior:*
 * Application should consume the record 1 more time, then move on to the next offset in the partition

*Actual behavior:*
 * Application will often work. Occasionally will use the offset from another partition and assign that to the partition where the record failed. This can then result in the consumer replaying messages instead of moving forward.

I put together a sample that can recreate the error. However, given its intermittent nature it may not fail on each run. I have included the logs from 3 different runs on my laptop from this test. Two of them show the error occurring. One of the them has a successful run. I have also provided more details in the README. 
 * [https://github.com/CodeSmell/CamelKafkaOffset]

This seems related to other issues with how Camel processes the _breakOnFirstError_ attribute. 
 * CAMEL-14935
 * CAMEL-18350
 * CAMEL-19894


> camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-20044
>                 URL: https://issues.apache.org/jira/browse/CAMEL-20044
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kafka
>    Affects Versions: 3.21.0
>         Environment: * Rocky Linux 8.7
>  * Open JDK 11.0.8
>  * Camel 3.21.0
>  * Spring Boot 2.7.14
>  * Strimzi Kafka 0.28.0/3.0.0
>            Reporter: Mike Barlotta
>            Priority: Major
>
> {*}Reproducing (intermittent){*}:
>  * Configure camel kafka consumer with following:
>  ** autoCommitEnable = false
>  ** allowManualCommit = true
>  ** autoOffsetReset = earliest
>  ** maxPollRecords = 1
>  ** breakOnFirstError = true
>  * Produce a series of records to kafka record to both partitions.
>  * Throw an exception that is unhandled
>  * commit the offset in the onException block
> *Expected behavior:*
>  * Application should consume the record 1 more time, then move on to the next offset in the partition
> *Actual behavior:*
>  * Application will often work. Occasionally will use the offset from another partition and assign that to the partition where the record failed. This can then result in the consumer replaying messages instead of moving forward.
> I put together a sample that can recreate the error. However, given its intermittent nature it may not fail on each run. I have included the logs from 3 different runs on my laptop from this test. Two of them show the error occurring. One of the them has a successful run. I have also provided more details in the README. 
>  * [https://github.com/CodeSmell/CamelKafkaOffset]
> This seems related to other issues with how Camel processes the _breakOnFirstError_ attribute. 
>  * CAMEL-17925
>  * CAMEL-14935
>  * CAMEL-18350
>  * CAMEL-19894



--
This message was sent by Atlassian Jira
(v8.20.10#820010)