[jira] [Created] (KAFKA-10280) Message filtering support based on keys format/Headers

["Raj (Jira)" <ji...@apache.org>]

Raj created KAFKA-10280:
---------------------------

             Summary: Message filtering support based on keys format/Headers
                 Key: KAFKA-10280
                 URL: https://issues.apache.org/jira/browse/KAFKA-10280
             Project: Kafka
          Issue Type: Improvement
          Components: consumer, core
            Reporter: Raj


There are many scenarios where there is a need for consumers to subscribe to a message based on pattern eg. Key format or headers. 
Typically, this scenario gets solved using external components implementing a filtering logic.

In a deployment with a large number of consumers,This becomes a huger performance (network/IO) overheads if most messages are getting discarded by the consumer based on the filter. If there are 10000's of consumers subscribing to the topic partitions, there is unnecessary IO, which could be avoided broker can apply filter per subscriptions.
 
I wanted to hear insights from the Kafka community around how they are solving this problem and gauge interest in formally submitting KIP around filtering as part of core capability.
 
Thanks,
Raj

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)