You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Narala, Ravi" <Ra...@Level3.com> on 2012/06/25 16:57:01 UTC
Consumer group and Partitions
Hi,
I have couple of questions regarding the concepts of Consumer group and partitions. Here are those ?
1. When consuming a message how Do I know which consumer group I belong to?. Is there a configuration available or do we have to specify it at the application level ?
2. How do consumer groups work in conjunction with partitions ?
3. How do I produce a message to specific group and a partition ?. A coding example is greatly appreciated.
Thanks,
Ravi Narala
Web Services & Integration
Level 3 Communications
200 Office Galleria Center
Southfield, MI 48035 US
p: 248-436-9324
e: ravi.narala@level3.com<ma...@level3.com>
Re: Consumer group and Partitions
Posted by Joel Koshy <jj...@gmail.com>.
You can just send it SIGTERM signal, or if you're running it off the
command-line for testing, ctrl-C will do a clean shutdown.
> Also my understanding is, If I send a message to a partition by specifying
> a partition key, all the messages sent from that producer will be sent to
> the same partition. If so how does a consumer get that message without
> knowing which partition the message was sent to. I know the consumer gets
> the messages from the topic by subscription. Are the messages replicated
> between the partitions? If so how ?
>
Right now, there is no custom partitioning on the consumer-side. Each
partition will be consumed by at most one consumer within a consumer group
(this is described at the end of the design document:
http://incubator.apache.org/kafka/design.html).
Thanks,
Joel
RE: Consumer group and Partitions
Posted by "Narala, Ravi" <Ra...@Level3.com>.
Thanks for the quick reply. How do we restart the broker. Is there a separate command?
Also my understanding is, If I send a message to a partition by specifying a partition key, all the messages sent from that producer will be sent to the same partition. If so how does a consumer get that message without knowing which partition the message was sent to. I know the consumer gets the messages from the topic by subscription. Are the messages replicated between the partitions? If so how ?
Ravi Narala
Web Services & Integration
Level 3 Communications
200 Office Galleria Center
Southfield, MI 48035 US
p: 248-436-9324
e: ravi.narala@level3.com
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Monday, June 25, 2012 1:48 PM
To: kafka-users@incubator.apache.org
Subject: Re: Consumer group and Partitions
That should be all you need - although you only need to bounce the kafka broker. Are you sending enough messages? If you send only a few test messages, it is possible for the messages to get sent to only one partition. E.g., I tried a quick test with the ConsoleProducer and it took three messages before the partition was created.
Joel
On Mon, Jun 25, 2012 at 8:43 AM, Narala, Ravi <Ra...@level3.com>wrote:
> Jun ,
> Thanks for the quick repsonse. I am running into a partition issue. I
> am not sure if am doing the right way. Here are the steps I did to
> create the partition. But I still do not see them partitioned.
>
> Stopped the kafka and zookeeper server Modified the num.partitions
> entry in server.properties from 1 to 2.
> Restarted the zookeeper and kafka server Used a Producer client to
> send messages to a topic named 'poc_test_topic1'.(I did not specify
> any partition key)
>
> The logs dir was configured to /tmp/kafka-logs in server.properties.
>
> I see only a 'poc_test_topic1-0' topic under logs dir. I was expecting
> to see 'poc_test_topic1-0' and 'poc_test_topic1-1' partitions under it.
>
> Am I missing something or is this all I need to create a partition ?
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: ravi.narala@level3.com
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Monday, June 25, 2012 11:05 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Consumer group and Partitions
>
> Ravi,
>
> For 1, consumer group is specified in consumer config.
>
> For 2, partitions are divided evenly among consumers in the same group
> for consumption. For details, please see the design doc on our website.
>
> For 3, a producer never needs to know consumer groups. A producer
> always produces messages to a topic/partition. There are examples in
> the design doc too.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <Ravi.Narala@level3.com
> >wrote:
>
> > Hi,
> > I have couple of questions regarding the concepts of Consumer group
> > and partitions. Here are those ?
> >
> >
> > 1. When consuming a message how Do I know which consumer group I
> > belong to?. Is there a configuration available or do we have to
> > specify it at the application level ?
> >
> > 2. How do consumer groups work in conjunction with partitions ?
> >
> > 3. How do I produce a message to specific group and a partition ?.
> A
> > coding example is greatly appreciated.
> >
> >
> >
> > Thanks,
> >
> > Ravi Narala
> > Web Services & Integration
> > Level 3 Communications
> > 200 Office Galleria Center
> > Southfield, MI 48035 US
> > p: 248-436-9324
> > e: ravi.narala@level3.com<ma...@level3.com>
> >
> >
>
Re: Consumer group and Partitions
Posted by Joel Koshy <jj...@gmail.com>.
That should be all you need - although you only need to bounce the kafka
broker. Are you sending enough messages? If you send only a few test
messages, it is possible for the messages to get sent to only one
partition. E.g., I tried a quick test with the ConsoleProducer and it took
three messages before the partition was created.
Joel
On Mon, Jun 25, 2012 at 8:43 AM, Narala, Ravi <Ra...@level3.com>wrote:
> Jun ,
> Thanks for the quick repsonse. I am running into a partition issue. I am
> not sure if am doing the right way. Here are the steps I did to create the
> partition. But I still do not see them partitioned.
>
> Stopped the kafka and zookeeper server
> Modified the num.partitions entry in server.properties from 1 to 2.
> Restarted the zookeeper and kafka server
> Used a Producer client to send messages to a topic named
> 'poc_test_topic1'.(I did not specify any partition key)
>
> The logs dir was configured to /tmp/kafka-logs in server.properties.
>
> I see only a 'poc_test_topic1-0' topic under logs dir. I was expecting to
> see 'poc_test_topic1-0' and 'poc_test_topic1-1' partitions under it.
>
> Am I missing something or is this all I need to create a partition ?
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: ravi.narala@level3.com
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Monday, June 25, 2012 11:05 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Consumer group and Partitions
>
> Ravi,
>
> For 1, consumer group is specified in consumer config.
>
> For 2, partitions are divided evenly among consumers in the same group for
> consumption. For details, please see the design doc on our website.
>
> For 3, a producer never needs to know consumer groups. A producer always
> produces messages to a topic/partition. There are examples in the design
> doc too.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <Ravi.Narala@level3.com
> >wrote:
>
> > Hi,
> > I have couple of questions regarding the concepts of Consumer group
> > and partitions. Here are those ?
> >
> >
> > 1. When consuming a message how Do I know which consumer group I
> > belong to?. Is there a configuration available or do we have to
> > specify it at the application level ?
> >
> > 2. How do consumer groups work in conjunction with partitions ?
> >
> > 3. How do I produce a message to specific group and a partition ?.
> A
> > coding example is greatly appreciated.
> >
> >
> >
> > Thanks,
> >
> > Ravi Narala
> > Web Services & Integration
> > Level 3 Communications
> > 200 Office Galleria Center
> > Southfield, MI 48035 US
> > p: 248-436-9324
> > e: ravi.narala@level3.com<ma...@level3.com>
> >
> >
>
RE: Consumer group and Partitions
Posted by "Narala, Ravi" <Ra...@Level3.com>.
Jun ,
Thanks for the quick repsonse. I am running into a partition issue. I am not sure if am doing the right way. Here are the steps I did to create the partition. But I still do not see them partitioned.
Stopped the kafka and zookeeper server
Modified the num.partitions entry in server.properties from 1 to 2.
Restarted the zookeeper and kafka server
Used a Producer client to send messages to a topic named 'poc_test_topic1'.(I did not specify any partition key)
The logs dir was configured to /tmp/kafka-logs in server.properties.
I see only a 'poc_test_topic1-0' topic under logs dir. I was expecting to see 'poc_test_topic1-0' and 'poc_test_topic1-1' partitions under it.
Am I missing something or is this all I need to create a partition ?
Ravi Narala
Web Services & Integration
Level 3 Communications
200 Office Galleria Center
Southfield, MI 48035 US
p: 248-436-9324
e: ravi.narala@level3.com
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Monday, June 25, 2012 11:05 AM
To: kafka-users@incubator.apache.org
Subject: Re: Consumer group and Partitions
Ravi,
For 1, consumer group is specified in consumer config.
For 2, partitions are divided evenly among consumers in the same group for consumption. For details, please see the design doc on our website.
For 3, a producer never needs to know consumer groups. A producer always produces messages to a topic/partition. There are examples in the design doc too.
Thanks,
Jun
On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <Ra...@level3.com>wrote:
> Hi,
> I have couple of questions regarding the concepts of Consumer group
> and partitions. Here are those ?
>
>
> 1. When consuming a message how Do I know which consumer group I
> belong to?. Is there a configuration available or do we have to
> specify it at the application level ?
>
> 2. How do consumer groups work in conjunction with partitions ?
>
> 3. How do I produce a message to specific group and a partition ?. A
> coding example is greatly appreciated.
>
>
>
> Thanks,
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: ravi.narala@level3.com<ma...@level3.com>
>
>
Re: Consumer group and Partitions
Posted by Jun Rao <ju...@gmail.com>.
Ravi,
For 1, consumer group is specified in consumer config.
For 2, partitions are divided evenly among consumers in the same group for
consumption. For details, please see the design doc on our website.
For 3, a producer never needs to know consumer groups. A producer always
produces messages to a topic/partition. There are examples in the design
doc too.
Thanks,
Jun
On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <Ra...@level3.com>wrote:
> Hi,
> I have couple of questions regarding the concepts of Consumer group and
> partitions. Here are those ?
>
>
> 1. When consuming a message how Do I know which consumer group I
> belong to?. Is there a configuration available or do we have to specify it
> at the application level ?
>
> 2. How do consumer groups work in conjunction with partitions ?
>
> 3. How do I produce a message to specific group and a partition ?. A
> coding example is greatly appreciated.
>
>
>
> Thanks,
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: ravi.narala@level3.com<ma...@level3.com>
>
>