You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Claus Ibsen (Jira)" <ji...@apache.org> on 2025/06/12 05:53:00 UTC

[jira] [Resolved] (CAMEL-22150) camel-kafka - Issue with Batching: Missed Records Due to due hasExpiredRecords check

     [ https://issues.apache.org/jira/browse/CAMEL-22150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-22150.
---------------------------------
    Resolution: Fixed

> camel-kafka - Issue with Batching: Missed Records Due to due hasExpiredRecords check
> ------------------------------------------------------------------------------------
>
>                 Key: CAMEL-22150
>                 URL: https://issues.apache.org/jira/browse/CAMEL-22150
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kafka
>    Affects Versions: 4.10.5, 4.11.0
>            Reporter: Adithya Kashyap H M
>            Priority: Major
>             Fix For: 4.10.6, 4.13.0
>
>
> When using the batching approach in Camel-Kafka, we observed an issue related to how polled {{ConsumerRecords}} and the {{exchangeList}} are handled.
> Inside the {{hasExpiredRecords}} method, two conditions are checked:
> {code:java}
> private boolean hasExpiredRecords(ConsumerRecords<Object, Object> consumerRecords) {
>     // no records in batch
>     if (exchangeList.isEmpty()) {
>         return false;
>     }
>     // timeout is only triggered if we no new records
>     boolean timeout = consumerRecords.isEmpty() && timeoutWatch.taken() >= configuration.getPollTimeoutMs();
>     // interval is triggered if enabled, and it has been X time since last batch completion
>     boolean interval = configuration.getBatchingIntervalMs() != null
>             && intervalWatch.taken() >= configuration.getBatchingIntervalMs();
>     return timeout || interval;
> } {code}
> If the {{intervalWatch}} exceeds the configured {{{}batchingIntervalMs{}}}, the condition evaluates to {{{}true{}}}. At this point, Camel proceeds to process the existing messages in the {{{}exchangeList{}}}.
> However, any new records fetched in the current Kafka poll ({{{}consumerRecords{}}}) are *not* *added* to the {{exchangeList}} for processing and it *returns* the result. Due to Kafka's polling behaviour, the next poll starts *after* the offset of these already-fetched-but-unprocessed records. As a result, these records are effectively skipped.
> If we continue to process and commit newly polled messages in the next iteration, the previously fetched (but unprocessed) messages are lost.
> This currently resulting in {*}data loss{*}, and we believe this is a critical issue in the batching logic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)