server-side message filter

[Xiangyuan LI <fl...@gmail.com>]

Hi,
 I want to know whether kafka has any plan to filter message on
server-side.this can reduce network flow obviously when many groups
subscribe the same one topic and only need part of message.

Re: server-side message filter

[Svante Karlsson <sv...@csi.se>]

The other problem would be that kafka has no idea of the serdes you use in
your messages ie it only sees the key and value as bytes so it would be
difficult to implement generic filtering in the broker.

for your second problem it seems that creating separate topics for
respective destinations solves your problem.

Another option would be to implement your own filtering proxy with some
other api (say streaming grpc) with knowledge of your data. And run this
close to your brokers. Zero-copy is out of the question and you get the
full traffic from broker to filtering proxy. We did this, but for a
multi-tenant use case where we did not want to expose kafka externaly but
still needed streaming.

/svante

>
>
>

Re: server-side message filter

[Xiangyuan LI <fl...@gmail.com>]

Hi,
  thx for reply.we have 2 situation to use server-side filter,that also
mean advantage consumer filter doen't have:
  1.as Guozhang said,server filter would reduce network cost. we use aws
ec2 to provide kafka, all of them and kafka clients in different aws
regions to guarantee availability. network flow cross regions means large
bill,sometimes even more expensive than several ec2 instances.
  2.not only aws, my employer use many public cloud services,and often need
use kafka to sync data from one cloud service to another.

  now we have some kinds of message,each of them has different tags like
messageType,deviceType,manufacturer,city ,and so on.and we have some
application,one need to analyze message that messageType=hearbeat &
manufacturer=microsoft ,one need analyze message that deviceType=switch &
messageType=userOperation. every application consume different tags
combination,it's
hard to decide send meessage to which topic and recive message from which
topic.

 I know kafka send message use zero-copy, if kafka implement  server
filter, its code may looks like LazyDownConversionRecords and will cause
some performance
problem for kafka,but we really think it's  necessary.

  thx.

Guozhang Wang <wa...@gmail.com> 于2019年8月28日周三 下午11:53写道：

> The reduction on network cost is a valid point: letting the consumer to do
> the filtering would waste the network bandwidth as many bytes would be
> dropped on the floor directly afterwards. But note that letting brokers to
> filter on fetch response means that we cannot use nio zero-copy as we need
> to do this filtering by allocating the memory in user space which could be
> a big burden at broker-side.
>
> Xianyuan: have you considered using different topics instead, i.e. having a
> finer-grained grouping of messages based on the consumer subscriptions?
>
> Guozhang
>
> On Wed, Aug 28, 2019 at 8:14 AM Adam Bellemare <ad...@gmail.com>
> wrote:
>
> > I do not think this will be implemented in the server. Why not simply
> > create a consumer that filters as required and outputs to a new topic?
> >
> > > On Aug 28, 2019, at 7:48 AM, Xiangyuan LI <fl...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > > I want to know whether kafka has any plan to filter message on
> > > server-side.this can reduce network flow obviously when many groups
> > > subscribe the same one topic and only need part of message.
> >
>
>
> --
> -- Guozhang
>

Re: server-side message filter

[Guozhang Wang <wa...@gmail.com>]

The reduction on network cost is a valid point: letting the consumer to do
the filtering would waste the network bandwidth as many bytes would be
dropped on the floor directly afterwards. But note that letting brokers to
filter on fetch response means that we cannot use nio zero-copy as we need
to do this filtering by allocating the memory in user space which could be
a big burden at broker-side.

Xianyuan: have you considered using different topics instead, i.e. having a
finer-grained grouping of messages based on the consumer subscriptions?

Guozhang

On Wed, Aug 28, 2019 at 8:14 AM Adam Bellemare <ad...@gmail.com>
wrote:

> I do not think this will be implemented in the server. Why not simply
> create a consumer that filters as required and outputs to a new topic?
>
> > On Aug 28, 2019, at 7:48 AM, Xiangyuan LI <fl...@gmail.com>
> wrote:
> >
> > Hi,
> > I want to know whether kafka has any plan to filter message on
> > server-side.this can reduce network flow obviously when many groups
> > subscribe the same one topic and only need part of message.
>


-- 
-- Guozhang

Re: server-side message filter

[Adam Bellemare <ad...@gmail.com>]

I do not think this will be implemented in the server. Why not simply create a consumer that filters as required and outputs to a new topic? 

> On Aug 28, 2019, at 7:48 AM, Xiangyuan LI <fl...@gmail.com> wrote:
> 
> Hi,
> I want to know whether kafka has any plan to filter message on
> server-side.this can reduce network flow obviously when many groups
> subscribe the same one topic and only need part of message.