You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Kip Kohn (Jira)" <ji...@apache.org> on 2021/09/20 18:39:00 UTC

[jira] [Created] (GOBBLIN-1549) GaaS should retry processing Kafka events in case of an error

Kip Kohn created GOBBLIN-1549:
---------------------------------

             Summary: GaaS should retry processing Kafka events in case of an error
                 Key: GOBBLIN-1549
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1549
             Project: Apache Gobblin
          Issue Type: Improvement
          Components: gobblin-service
            Reporter: Kip Kohn
            Assignee: Abhishek Tiwari


[per  [https://github.com/aplex] ] If we cannot process a Kafka event in GaaS, the processing thread will shutdown, and messages from that Kafka partition will never be processed.

Kafka message can fail to get processed for multiple reasons. Some of them can be transient, like network errors when connecting to the database. Others can be more-or-less permanent, like invalid message.

We should retry processing the messages with exponential backoff.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)