Dead letter queue support in Kafka

[Alexander Pakulov <a....@gmail.com>]

Hi everyone,

Since Kafka doesn’t have a dead-letter queue support built in - I’m looking for advice and best approaches to handle bad messages or cases when system is going crazy, once you receive an exception it basically means you’re blocking the whole kaka-stream from consuming other messages in the partition.

There are couple of ideas that potentially could address the problem:
1) Wrap message processing logging into try-catch block -> log the error if there is an exception and proceed to the next messages. Which basically mean just swallow the message.
2) Dead-letter queue could be implemented on top of Kafka itself, the same try-catch block -> if there is an exception you forward broken message to dead-letter topic and consume it later once all issues are resolved.

Any feedback and suggestions are highly welcomed.

Thank you!
-- 
Alexander Pakulov