[jira] [Created] (CAMEL-15228) Missing end of polling signal

["Jörg (Jira)" <ji...@apache.org>]

Jörg created CAMEL-15228:
----------------------------

             Summary: Missing end of polling signal
                 Key: CAMEL-15228
                 URL: https://issues.apache.org/jira/browse/CAMEL-15228
             Project: Camel
          Issue Type: New Feature
          Components: camel-kafka
    Affects Versions: 3.4.0
            Reporter: Jörg


For batch requirements to manual commit all partitions at the end of a pull request to some topics, it is currently not possible to get a signal for the latest result within a poll request in processors.

The problem with this is, that a Processor cannot know when the last record of a poll received and it is time for commits to all of the affected partitions, because you cannot be sure that there is always a content in every partition.

Solution could be to add a property to the manual commit handler in the [KafkaConsumer.KafkaFetchRecords.doRun|[https://github.com/apache/camel/blob/cca5369b2308cd2a5d35ebb98a092dbc73406d54/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java#L336]]

 
{code:java}
exchange.getIn().setHeader(KafkaConstants.MANUAL_COMMIT, manual);

//adding new exchange property
exchange.getIn().setHeader(KafkaConstants.LAST_RECORD_OF_CURRENT_POLL_REQUEST, last_record_of_last_partition);
{code}
LAST_RECORD_IN_CURRENT_POLL_REQUEST should be a boolean to signal end of the [loop over all partitions|https://github.com/apache/camel/blob/c86c860b7c5b5ef754970291b7ad7c4b32b443c6/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java#L313]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)