You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kumar Jayanti <ku...@gmail.com> on 2015/05/26 18:51:39 UTC
Ordered Message Queue with Pool of Consumers
Our system has a specific message type that requires total ordering.
Message-1 should be consumed (acted upon) before Message-2 is consumed.
However since we have a cluster of consumer nodes we would like to
round-robin the processing of messages if possible to different nodes while
maintaining the ordering. Is this possible ?
>From my reading it appears we have to pin it down to a Topic with Single
Partition with Single Consumer Process but i may be wrong ?.
Re: Ordered Message Queue with Pool of Consumers
Posted by Jason Rosenberg <jb...@squareup.com>.
How would you apply total ordering if multiple messages are being consumed
in parallel? If message-1 and message-2 are being consumed in parallel,
do you really mean you want to guarantee that message-1 is consumed before
the consumption of message-2 begins?
On Tue, May 26, 2015 at 1:34 PM, Kumar Jayanti <ku...@gmail.com>
wrote:
> Thanks for your response.
>
> The requirement is to have total ordering across partitions. I do have a
> Key and all my consumers would be in the same consumer group. I am new to
> kafka and so would like to reconfirm my understanding of your last
> statement :
>
> "So assuming you have a key you can assign to organize them, you should be
> able to use many topic partitions and consumers."
>
> Let me explore a bit more on your suggestion and come back if i have
> further doubts.
>
> thanks.
>
>
>
>
>
>
>
> On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> wrote:
>
> > Messages are consumed in order from each topic partition. You can make
> sure
> > messages end up in the same topic partition by making them have the same
> > key, or by explicitly mapping them to partitions yourself when producing
> > them.
> >
> > To parallelize consumption, you use a consumer group. Each time you
> create
> > a consumer, you can specify the group it should be in and work is
> > automatically balanced across all the members of the group. This is done
> on
> > a per topic-partition basis, so when the consumer group membership is
> > stable, a single consumer will see all the messages from a given topic
> > partition in order.
> >
> > So assuming you have a key you can assign to organize them, you should be
> > able to use many topic partitions and consumers.
> >
> > On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti <ku...@gmail.com>
> > wrote:
> >
> > > Our system has a specific message type that requires total ordering.
> > > Message-1 should be consumed (acted upon) before Message-2 is consumed.
> > > However since we have a cluster of consumer nodes we would like to
> > > round-robin the processing of messages if possible to different nodes
> > while
> > > maintaining the ordering. Is this possible ?
> > >
> > > From my reading it appears we have to pin it down to a Topic with
> Single
> > > Partition with Single Consumer Process but i may be wrong ?.
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
Re: Ordered Message Queue with Pool of Consumers
Posted by Kumar Jayanti <ku...@gmail.com>.
Thanks for your response.
The requirement is to have total ordering across partitions. I do have a
Key and all my consumers would be in the same consumer group. I am new to
kafka and so would like to reconfirm my understanding of your last
statement :
"So assuming you have a key you can assign to organize them, you should be
able to use many topic partitions and consumers."
Let me explore a bit more on your suggestion and come back if i have
further doubts.
thanks.
On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Messages are consumed in order from each topic partition. You can make sure
> messages end up in the same topic partition by making them have the same
> key, or by explicitly mapping them to partitions yourself when producing
> them.
>
> To parallelize consumption, you use a consumer group. Each time you create
> a consumer, you can specify the group it should be in and work is
> automatically balanced across all the members of the group. This is done on
> a per topic-partition basis, so when the consumer group membership is
> stable, a single consumer will see all the messages from a given topic
> partition in order.
>
> So assuming you have a key you can assign to organize them, you should be
> able to use many topic partitions and consumers.
>
> On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti <ku...@gmail.com>
> wrote:
>
> > Our system has a specific message type that requires total ordering.
> > Message-1 should be consumed (acted upon) before Message-2 is consumed.
> > However since we have a cluster of consumer nodes we would like to
> > round-robin the processing of messages if possible to different nodes
> while
> > maintaining the ordering. Is this possible ?
> >
> > From my reading it appears we have to pin it down to a Topic with Single
> > Partition with Single Consumer Process but i may be wrong ?.
> >
>
>
>
> --
> Thanks,
> Ewen
>
Re: Ordered Message Queue with Pool of Consumers
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Messages are consumed in order from each topic partition. You can make sure
messages end up in the same topic partition by making them have the same
key, or by explicitly mapping them to partitions yourself when producing
them.
To parallelize consumption, you use a consumer group. Each time you create
a consumer, you can specify the group it should be in and work is
automatically balanced across all the members of the group. This is done on
a per topic-partition basis, so when the consumer group membership is
stable, a single consumer will see all the messages from a given topic
partition in order.
So assuming you have a key you can assign to organize them, you should be
able to use many topic partitions and consumers.
On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti <ku...@gmail.com>
wrote:
> Our system has a specific message type that requires total ordering.
> Message-1 should be consumed (acted upon) before Message-2 is consumed.
> However since we have a cluster of consumer nodes we would like to
> round-robin the processing of messages if possible to different nodes while
> maintaining the ordering. Is this possible ?
>
> From my reading it appears we have to pin it down to a Topic with Single
> Partition with Single Consumer Process but i may be wrong ?.
>
--
Thanks,
Ewen