[jira] [Updated] (CAMEL-16105) AllowManualCommit does not ensure full control of the commit flows

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-16105:
--------------------------------
    Fix Version/s: 3.10.0

> AllowManualCommit does not ensure full control of the commit flows
> ------------------------------------------------------------------
>
>                 Key: CAMEL-16105
>                 URL: https://issues.apache.org/jira/browse/CAMEL-16105
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-kafka
>    Affects Versions: 3.7.1
>            Reporter: Dmitrijs Jesilevskis
>            Priority: Major
>             Fix For: 3.10.0
>
>
> *Scenario*
> My scenario implies handling business-critical data through Kafka, before writing it to the sink system. To ensure maximum possible delivery guarantee and control, I choose to use Kafka component with _allowManualCommit_ option, as per [documentation|https://camel.apache.org/components/latest/kafka-component.html#_using_manual_commit_with_kafka_consumer]. This is required to be absolutely sure that message is delivered to the sink system, before committing the state and move to the next item or batch of items.
> *Problem*
>  When I use _allowManualCommit_ option but do not (intentionally) include any steps to actually execute the commit, the Consumer Group offset is still being committed, when I restart integrations.
>  The commit is executed due to triggering of _onPartitionsRevoked_ event handler in [KafkaConsumer.java, L473|https://github.com/apache/camel/blob/0dcbe44d35545a4fc29e5e7b0e9e687d564cbf28/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java#L473]. This happens due to graceful termination of the running container.
>  * Line 485, initiates _commitOffset()_ method with _forceCommit_ parameter set to true;
>  * Line 453 host a condition that executes offset persistence to custom offsetRepository when _allowManualCommit_ is disabled, and offsetRepository is specified - this condition is skipped, as in my scenario _allowManualCommit_ is enabled
>  * Line 458 host an alternative condition to execute a commit when _forceCommit_ parameter is _true_, and it is indeed true, as the method is invoked by _onPartitionsRevoked_.
> The hidden side-effect of the scenario is that offset is committed, despite being ordered. This might potentially lead to the loss of messages, as there is no guarantee that messages were processed successfully.
> *Possible solution*
>  * Execute _forceCommit_ section only when _allowManualCommit_ is disabled. In case of a manual commit, the developer should be in charge in all cases.
> *Extra*
> It must be also ensured that when custom offsetRepository is used, state is never persisted to both custom repository and standard consumer group. Currently that what happens when offsetRepository is paired with enabled _allowManualCommit_, through the above described scenario as well.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)