Kafka handle errors when consuming messages

[Prachi Wadekar <pr...@gmail.com>]

Hi,

I have a question about handling errors once a message has been consumed. I
have a system where, the consumer consumes messages and tries to write the
data to a database. These writes should happen serially.

There are scenarios where there could be errors doing the downstream write,
which unfortunately cannot be fixed by retrying. They may require manual
intervention. Thus these messages are inserted in another DLQ topic. Once
the issue is resolved, I would need another consumer to consume an errored
message from the DLQ topic, retry inserting that into the database and
replay all subsequent events after this message in the original topic.

What is the best way to do this?

Thanks,
Prachi