[jira] [Commented] (KAFKA-1448) Filter-plugins for messages

["Neha Narkhede (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996558#comment-13996558 ] 

Neha Narkhede commented on KAFKA-1448:
--------------------------------------

>> Using one topic for each event * product combination would yield around 2000 topics, which seems to be not what kafka is designed for.

In general, 2000 topics should work fine in Kafka. But that depends on what the growth rate is beyond that. You can achieve filtering by writing a wrapper around your Kafka consumer that returns messages that pass the filter. There isn't much advantage to exposing this in Kafka APIs since there are no IO/CPU savings that can be achieved by doing that. 

> Filter-plugins for messages
> ---------------------------
>
>                 Key: KAFKA-1448
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1448
>             Project: Kafka
>          Issue Type: Wish
>          Components: consumer
>            Reporter: Moritz Möller
>            Assignee: Neha Narkhede
>
> Hi,
> we use Kafka to transmit different events that occur on different products, and would like to be able to subscribe only to a certain set of events for certain products.
> Using one topic for each event * product combination would yield around 2000 topics, which seems to be not what kafka is designed for.
> What we would need is a way to add a consumer filter plugin to kafka (a simple class that can accept or reject a message) and to pass a parameter from the consumer to that filter class.
> Is there a better way to do this already, or if not, would you accept a patch upstream that adds such a mechanism?
> Thanks,
> Mo



--
This message was sent by Atlassian JIRA
(v6.2#6252)