[jira] [Created] (NIFI-13787) ConsumeKafka does not provide proper error handling

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIFI-13787:
---------------------------------

             Summary: ConsumeKafka does not provide proper error handling
                 Key: NIFI-13787
                 URL: https://issues.apache.org/jira/browse/NIFI-13787
             Project: Apache NiFi
          Issue Type: Bug
          Components: Extensions
    Affects Versions: 2.0.0-M4, 2.0.0-M3, 2.0.0-M2
            Reporter: Mark Payne
            Assignee: Mark Payne
             Fix For: 2.0.0-M5


ConsumeKafka does not properly handle failure conditions.

In the event that an Exception is thrown, the consumer never rolls back the offsets that have been consumed. As a result, this means that the next time the processor runs, it will consume the next batch of events. If this is successful, the commits when then be committed, having lost the group of records that were previously consumed and not handled successfully.

This will likely require a rework of the abstractions that are provided by the Kafka Connection Service / Kafka Consumer Service. The consumer service attempts to handle connection pooling, but this means that the processor does not have the necessary access to a particular Consumer in order to rollback transactions, etc. Additionally, it leads to code that is rather complex, whereas the pooling can be sufficiently handled in a couple lines of code in the processor by simply using a BlockingQueue of consumers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)