You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Zeeshan Ali <Ze...@ibm.com> on 2021/05/12 20:22:25 UTC

Anyway to suppress duplicate Kafka logs



Hello,

There are times when Kafka would dump log messages at a rate such as 5
messages per second or 3 messages per second which are duplicated every
second. I have tried BurstFilter in log4j2 however it can swallow messages.
I am considering usage of Script filter of log4j2 but haven't tried it yet,
not sure if it will be able to filter duplicate messages without swallowing
any log messages. Thought I would turn to Kafka experts If they have a
known solution or if there is any support to achieve this in Kafka?

Thanks in Advance!!

Zeeshan