[jira] [Commented] (KAFKA-3895) Implement a default queue for expired messages

["Jeff Klukas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533221#comment-15533221 ] 

Jeff Klukas commented on KAFKA-3895:
------------------------------------

It's not clear to me how the concepts here map onto the Kafka consumer model.

What do you consider an "expired" message?

I've been in some discussions recently with colleagues considering how we might get dead letter-like functionality in Kafka. In our case, we're usually concerned with a badly formed message that doesn't deserialize correctly or otherwise causes a non-retriable error in the consumer.

An idea we have considered is creating a single deadletter topic for the cluster. When a consumer gets an unretriable error for a message, it would produce a message to that deadletter topic where the key would be group.id and the value would be a tuple of (topic, partition, offset) that failed to be consumed. We'd then be able to go back within the retention period, figure out why the message failed to be consumed, change our consumer logic, and reconsume that particular message if needed.

> Implement a default queue for expired messages
> ----------------------------------------------
>
>                 Key: KAFKA-3895
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3895
>             Project: Kafka
>          Issue Type: New Feature
>          Components: core
>            Reporter: Krish Iyer
>            Priority: Minor
>
> As discussed in the mailing list, kafka currently does not support a dead letter queue-like feature where all the expired messages can be sent to.
> A high-level design of such a feature can be as follows:
> When kafka needs to expire a message from topic 'X', it sends the message to 'X-expired' topic, rather than deleting it. If 'X' is 'X-expired', dequeue and requeue the message in the expired topic.
> Given that this topic can receive duplicate messages (same expired message multiple times), the consumer of this topic needs to ensure duplicate message handling.
> Open question:
> Does the message need a timestamp to figure out when did it move from a normal queue to a dead letter queue. Does it need a flag to specify whether it had expired from the X-expired topic itself?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)