[jira] [Resolved] (KAFKA-9285) Implement failed message topic to account for processing lag during failure

["Richard Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Yu resolved KAFKA-9285.
-------------------------------
    Resolution: Fixed

Already resolved by Kafka Connect.

> Implement failed message topic to account for processing lag during failure
> ---------------------------------------------------------------------------
>
>                 Key: KAFKA-9285
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9285
>             Project: Kafka
>          Issue Type: New Feature
>          Components: consumer
>            Reporter: Richard Yu
>            Assignee: Richard Yu
>            Priority: Major
>              Labels: kip
>
> Presently, in current Kafka failure schematics, when a consumer crashes, the user is typically responsible for both detecting as well as restarting the failed consumer. Therefore, during this period of time, when the consumer is dead, it would result in a period of inactivity where no records are consumed, hence lag results. Previously, there has been attempts to resolve this problem: when failure is detected by broker, a substitute consumer will be started (the so-called [Rebalance Consumer|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-333%3A+Add+faster+mode+of+rebalancing]]) which will continue processing records in Kafka's stead. 
> However, this has complications, as records will only be stored locally, and in case of this consumer failing as well, that data will be lost. Instead, we need to consider how we can still process these records and at the same time effectively _persist_ them. It is here that I propose the concept of a _failed message topic._ At a high level, it works like this. When we find that a consumer has failed, messages which was originally meant to be sent to that consumer would be redirected to this failed messaged topic. The user can choose to assign consumers to this topic, which would consume messages (that would've originally been processed by the failed consumers) from it.
> Naturally, records from different topics can not go into the same failed message topic, since we cannot tell which records belong to which consumer.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)