RPC on top of kafka

[Pierre-Yves ritschard <py...@smallrivers.com>]

Hi,

One of the standard use cases for messaging queues is to implement some
sort of distributed RPC.

Let's pretend you have a bunch of worker processes, all registered in
the same consumer group (to make sure only one is done at a time) and
which are registered on a queue named to represent the task they know
how to do.

RPC could then be implemented by a dynamic consumer group creation and
registration on the client side, then sending a message containing the
created id in the payload, so that the worker process knows who to
respond to.

What I'm worried about is the overhead of registering new consumer
groups on the fly, would that put an anormal burden on the brokers ?

  - pyr

Re: RPC on top of kafka

[Eric Hauser <ew...@gmail.com>]

There are a things you might want to consider before using Kafka for
your use case:

1. Kafka producers currently do not get an acknowledgement, which may
or may not be a big deal for you
2. No messages will show up at your consumers until they hit disk (and
Kafka will keep all your request/response data on disk until the
retention period is reached).
3. Kafka is not really designed for low latency messaging.    You can
toggle some of the flush settings per topic, but you won't get the
same performance as something that is queuing in memory only.

On Thu, Sep 29, 2011 at 11:40 AM, Pierre-Yves ritschard
<py...@smallrivers.com> wrote:
> Hi,
>
> It would be short lived consumer sessions: setup consumer + consumer
> group, produce to worker, accept response, teardown consumer. So it
> would stay well beneath the maxopenfile threshold.
>
> What I was worried about is the potential overhead of creating and
> tearing down consumer groups since it generates zookeeper interaction.
>
>  - pyr
>
> On Thu, 29 Sep 2011 08:35:33 -0700
> Jun Rao <ju...@gmail.com> wrote:
>
>> Pierres-Yves,
>>
>> Since each consumer maintains its own state, there is relatively
>> little overhead per consumer on the broker. However, each consumer
>> has to maintain a socket connection to a broker, you can't have
>> infinite number of consumers. How many concurrent consumer groups do
>> you plan to have?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard
>> <py...@smallrivers.com>wrote:
>>
>> > Hi,
>> >
>> > One of the standard use cases for messaging queues is to implement
>> > some sort of distributed RPC.
>> >
>> > Let's pretend you have a bunch of worker processes, all registered
>> > in the same consumer group (to make sure only one is done at a
>> > time) and which are registered on a queue named to represent the
>> > task they know how to do.
>> >
>> > RPC could then be implemented by a dynamic consumer group creation
>> > and registration on the client side, then sending a message
>> > containing the created id in the payload, so that the worker
>> > process knows who to respond to.
>> >
>> > What I'm worried about is the overhead of registering new consumer
>> > groups on the fly, would that put an anormal burden on the brokers ?
>> >
>> >  - pyr
>> >
>
>

Re: RPC on top of kafka

[Pierre-Yves ritschard <py...@smallrivers.com>]

Hi,

It would be short lived consumer sessions: setup consumer + consumer
group, produce to worker, accept response, teardown consumer. So it
would stay well beneath the maxopenfile threshold.

What I was worried about is the potential overhead of creating and
tearing down consumer groups since it generates zookeeper interaction.

  - pyr

On Thu, 29 Sep 2011 08:35:33 -0700
Jun Rao <ju...@gmail.com> wrote:

> Pierres-Yves,
> 
> Since each consumer maintains its own state, there is relatively
> little overhead per consumer on the broker. However, each consumer
> has to maintain a socket connection to a broker, you can't have
> infinite number of consumers. How many concurrent consumer groups do
> you plan to have?
> 
> Thanks,
> 
> Jun
> 
> On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard
> <py...@smallrivers.com>wrote:
> 
> > Hi,
> >
> > One of the standard use cases for messaging queues is to implement
> > some sort of distributed RPC.
> >
> > Let's pretend you have a bunch of worker processes, all registered
> > in the same consumer group (to make sure only one is done at a
> > time) and which are registered on a queue named to represent the
> > task they know how to do.
> >
> > RPC could then be implemented by a dynamic consumer group creation
> > and registration on the client side, then sending a message
> > containing the created id in the payload, so that the worker
> > process knows who to respond to.
> >
> > What I'm worried about is the overhead of registering new consumer
> > groups on the fly, would that put an anormal burden on the brokers ?
> >
> >  - pyr
> >

Re: RPC on top of kafka

[Jun Rao <ju...@gmail.com>]

Pierres-Yves,

Since each consumer maintains its own state, there is relatively little
overhead per consumer on the broker. However, each consumer has to maintain
a socket connection to a broker, you can't have infinite number of
consumers. How many concurrent consumer groups do you plan to have?

Thanks,

Jun

On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard
<py...@smallrivers.com>wrote:

> Hi,
>
> One of the standard use cases for messaging queues is to implement some
> sort of distributed RPC.
>
> Let's pretend you have a bunch of worker processes, all registered in
> the same consumer group (to make sure only one is done at a time) and
> which are registered on a queue named to represent the task they know
> how to do.
>
> RPC could then be implemented by a dynamic consumer group creation and
> registration on the client side, then sending a message containing the
> created id in the payload, so that the worker process knows who to
> respond to.
>
> What I'm worried about is the overhead of registering new consumer
> groups on the fly, would that put an anormal burden on the brokers ?
>
>  - pyr
>