You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Heath Ivie <hi...@AutoAnything.com> on 2016/01/07 21:54:55 UTC
Kafka-Rest question
Hi,
I am building my application using the rest proxy and I wanted to get some clarification on the documentation.
"Because consumers are stateful, any consumer instances created with the REST API are tied to a specific REST proxy instance. A full URL is provided when the instance is created and it should be used to construct any subsequent requests. Failing to use the returned URL for future consumer requests will end up adding new consumers to the group. If a REST proxy instance is shutdown, it will attempt to cleanly destroy any consumers before it is terminated."
Does the highlighted mean that I have to have multiple deployed instances in order to have more than 1 consumer group and configure it with sticky sessions?
I have the following code where I am testing the behavior when I have 2 consumers in the same group, and I am getting an empty array back for the second consumer in the same group.
This will through an exception when I attempt to read a message from the 2nd consumer.
Not sure if it would help, but I configured the proxy to have 2 threads per consumer.
KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("http://10.1.30.48:8082");
KafkaMessageQueueReader reader1 = new KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() { ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest", ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
KafkaMessageQueueReader reader2 = new KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() { ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest", ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
KafkaMessageQueueReader reader3 = new KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() { ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest", ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
for (int i = 0; i < 10; i++)
{
//Publish a message to the ShellComparisonTest topic
writer.PublishMessage("ShellComparisonTest", new OrderEvent() { OrderId = 5, OrderItemId = 1234 });
var @event = reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
if (!@event.Any()) throw new Exception();
@event = reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
if (!@event.Any()) throw new Exception();
//Publish a message to the ShellComparisonTest topic so that
//there is one unread message in the queue before trying to
//read from the second consumer in the group
writer.PublishMessage("ShellComparisonTest", new OrderEvent() { OrderId = 5, OrderItemId = 1234 });
@event = reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
if (!@event.Any()) throw new Exception();
}
Is there something that I am missing or is there a good resource to read.
Thanks
Heath
Warning: This e-mail may contain information proprietary to AutoAnything Inc. and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you have received this message in error and any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete all copies.
RE: Kafka-Rest question
Posted by Heath Ivie <hi...@AutoAnything.com>.
That cool, I will check that out but it makes sense
-----Original Message-----
From: Ewen Cheslack-Postava [mailto:ewen@confluent.io]
Sent: Thursday, January 07, 2016 8:14 PM
To: users@kafka.apache.org
Subject: Re: Kafka-Rest question
The REST proxy doesn't currently handle topic creation, so you need to either do that separately or set the num.partitions config for your brokers if you are relying on topic auto creation. The default for num.partitions is 1, which is probably why you're seeing the current behavior.
It'd be great to add admin functionality to the proxy so you could do this directly (and KIP-4 should make that easier to accomplish).
-Ewen
On Thu, Jan 7, 2016 at 1:46 PM, Heath Ivie <hi...@autoanything.com> wrote:
> I think that the question I should be asking is how do I get more than
> 1 partition when using the rest client?
>
> -----Original Message-----
> From: Heath Ivie [mailto:hivie@AutoAnything.com]
> Sent: Thursday, January 07, 2016 1:28 PM
> To: users@kafka.apache.org
> Subject: RE: Kafka-Rest question
>
> Thanks Ewen.
>
> I have that in my example, but when there are two consumers each with
> an autogenerated id, only one can read and get data.
>
> The second one will return an empty array, even with a message being
> published after the first consumer is finished.
>
> I see what is expected if I use the .NET client, but in the proxy it
> doesn’t appear to work like that.
>
> -----Original Message-----
> From: Ewen Cheslack-Postava [mailto:ewen@confluent.io]
> Sent: Thursday, January 07, 2016 1:18 PM
> To: users@kafka.apache.org
> Subject: Re: Kafka-Rest question
>
> On Thu, Jan 7, 2016 at 12:54 PM, Heath Ivie <hi...@autoanything.com>
> wrote:
>
> > Hi,
> >
> > I am building my application using the rest proxy and I wanted to
> > get some clarification on the documentation.
> >
> > "Because consumers are stateful, any consumer instances created with
> > the REST API are tied to a specific REST proxy instance. A full URL
> > is provided when the instance is created and it should be used to
> > construct any subsequent requests. Failing to use the returned URL
> > for future consumer requests will end up adding new consumers to the
> > group. If a REST proxy instance is shutdown, it will attempt to
> > cleanly destroy any consumers before it is terminated."
> >
> > Does the highlighted mean that I have to have multiple deployed
> > instances in order to have more than 1 consumer group and configure
> > it with sticky sessions?
> >
>
> You can run multiple consumer instances in one or more consumer groups
> on a single REST proxy instance. However, you'll need to ensure they
> have different IDs (which you can have auto-assigned if you like).
>
>
> >
> > I have the following code where I am testing the behavior when I
> > have
> > 2 consumers in the same group, and I am getting an empty array back
> > for the second consumer in the same group.
> >
> > This will through an exception when I attempt to read a message from
> > the 2nd consumer.
> >
> > Not sure if it would help, but I configured the proxy to have 2
> > threads per consumer.
> >
> > KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("
> > http://10.1.30.48:8082");
> >
> >
> >
> > KafkaMessageQueueReader reader1 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> > KafkaMessageQueueReader reader2 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> > KafkaMessageQueueReader reader3 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48"
> > });
> >
> >
> >
> > for (int i = 0; i < 10; i++)
> > {
> > //Publish a message to the ShellComparisonTest topic
> > writer.PublishMessage("ShellComparisonTest", new
> > OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> > var @event =
> > reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> >
> > @event =
> > reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> >
> > //Publish a message to the ShellComparisonTest topic
> > so that
> > //there is one unread message in the queue before
> > trying
> to
> > //read from the second consumer in the group
> > writer.PublishMessage("ShellComparisonTest", new
> > OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> > @event =
> > reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> > }
> >
> > Is there something that I am missing or is there a good resource to read.
> >
> >
> How many partitions does your topic have? Did you create it before
> this test or was it auto-created? One reason you could see this
> behavior is if you only have 1 partition. In that case only 1 consumer
> would see any messages since only 1 consumer would have any partitions assigned to it.
>
> -Ewen
>
>
>
> > Thanks
> >
> > Heath
> >
> >
> > Warning: This e-mail may contain information proprietary to
> > AutoAnything Inc. and is intended only for the use of the intended
> > recipient(s). If the reader of this message is not the intended
> > recipient(s), you have received this message in error and any
> > review, dissemination, distribution or copying of this message is
> > strictly prohibited. If you have received this message in error,
> > please notify the sender immediately and delete all copies.
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
Thanks,
Ewen
Re: Kafka-Rest question
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
The REST proxy doesn't currently handle topic creation, so you need to
either do that separately or set the num.partitions config for your brokers
if you are relying on topic auto creation. The default for num.partitions
is 1, which is probably why you're seeing the current behavior.
It'd be great to add admin functionality to the proxy so you could do this
directly (and KIP-4 should make that easier to accomplish).
-Ewen
On Thu, Jan 7, 2016 at 1:46 PM, Heath Ivie <hi...@autoanything.com> wrote:
> I think that the question I should be asking is how do I get more than 1
> partition when using the rest client?
>
> -----Original Message-----
> From: Heath Ivie [mailto:hivie@AutoAnything.com]
> Sent: Thursday, January 07, 2016 1:28 PM
> To: users@kafka.apache.org
> Subject: RE: Kafka-Rest question
>
> Thanks Ewen.
>
> I have that in my example, but when there are two consumers each with an
> autogenerated id, only one can read and get data.
>
> The second one will return an empty array, even with a message being
> published after the first consumer is finished.
>
> I see what is expected if I use the .NET client, but in the proxy it
> doesn’t appear to work like that.
>
> -----Original Message-----
> From: Ewen Cheslack-Postava [mailto:ewen@confluent.io]
> Sent: Thursday, January 07, 2016 1:18 PM
> To: users@kafka.apache.org
> Subject: Re: Kafka-Rest question
>
> On Thu, Jan 7, 2016 at 12:54 PM, Heath Ivie <hi...@autoanything.com>
> wrote:
>
> > Hi,
> >
> > I am building my application using the rest proxy and I wanted to get
> > some clarification on the documentation.
> >
> > "Because consumers are stateful, any consumer instances created with
> > the REST API are tied to a specific REST proxy instance. A full URL is
> > provided when the instance is created and it should be used to
> > construct any subsequent requests. Failing to use the returned URL for
> > future consumer requests will end up adding new consumers to the
> > group. If a REST proxy instance is shutdown, it will attempt to
> > cleanly destroy any consumers before it is terminated."
> >
> > Does the highlighted mean that I have to have multiple deployed
> > instances in order to have more than 1 consumer group and configure it
> > with sticky sessions?
> >
>
> You can run multiple consumer instances in one or more consumer groups on
> a single REST proxy instance. However, you'll need to ensure they have
> different IDs (which you can have auto-assigned if you like).
>
>
> >
> > I have the following code where I am testing the behavior when I have
> > 2 consumers in the same group, and I am getting an empty array back
> > for the second consumer in the same group.
> >
> > This will through an exception when I attempt to read a message from
> > the 2nd consumer.
> >
> > Not sure if it would help, but I configured the proxy to have 2
> > threads per consumer.
> >
> > KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("
> > http://10.1.30.48:8082");
> >
> >
> >
> > KafkaMessageQueueReader reader1 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> > KafkaMessageQueueReader reader2 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> > KafkaMessageQueueReader reader3 = new
> > KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> > ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> > ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48"
> > });
> >
> >
> >
> > for (int i = 0; i < 10; i++)
> > {
> > //Publish a message to the ShellComparisonTest topic
> > writer.PublishMessage("ShellComparisonTest", new
> > OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> > var @event =
> > reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> >
> > @event =
> > reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> >
> > //Publish a message to the ShellComparisonTest topic
> > so that
> > //there is one unread message in the queue before trying
> to
> > //read from the second consumer in the group
> > writer.PublishMessage("ShellComparisonTest", new
> > OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> > @event =
> > reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
> > if (!@event.Any()) throw new Exception();
> > }
> >
> > Is there something that I am missing or is there a good resource to read.
> >
> >
> How many partitions does your topic have? Did you create it before this
> test or was it auto-created? One reason you could see this behavior is if
> you only have 1 partition. In that case only 1 consumer would see any
> messages since only 1 consumer would have any partitions assigned to it.
>
> -Ewen
>
>
>
> > Thanks
> >
> > Heath
> >
> >
> > Warning: This e-mail may contain information proprietary to
> > AutoAnything Inc. and is intended only for the use of the intended
> > recipient(s). If the reader of this message is not the intended
> > recipient(s), you have received this message in error and any review,
> > dissemination, distribution or copying of this message is strictly
> > prohibited. If you have received this message in error, please notify
> > the sender immediately and delete all copies.
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
Thanks,
Ewen
RE: Kafka-Rest question
Posted by Heath Ivie <hi...@AutoAnything.com>.
I think that the question I should be asking is how do I get more than 1 partition when using the rest client?
-----Original Message-----
From: Heath Ivie [mailto:hivie@AutoAnything.com]
Sent: Thursday, January 07, 2016 1:28 PM
To: users@kafka.apache.org
Subject: RE: Kafka-Rest question
Thanks Ewen.
I have that in my example, but when there are two consumers each with an autogenerated id, only one can read and get data.
The second one will return an empty array, even with a message being published after the first consumer is finished.
I see what is expected if I use the .NET client, but in the proxy it doesn’t appear to work like that.
-----Original Message-----
From: Ewen Cheslack-Postava [mailto:ewen@confluent.io]
Sent: Thursday, January 07, 2016 1:18 PM
To: users@kafka.apache.org
Subject: Re: Kafka-Rest question
On Thu, Jan 7, 2016 at 12:54 PM, Heath Ivie <hi...@autoanything.com> wrote:
> Hi,
>
> I am building my application using the rest proxy and I wanted to get
> some clarification on the documentation.
>
> "Because consumers are stateful, any consumer instances created with
> the REST API are tied to a specific REST proxy instance. A full URL is
> provided when the instance is created and it should be used to
> construct any subsequent requests. Failing to use the returned URL for
> future consumer requests will end up adding new consumers to the
> group. If a REST proxy instance is shutdown, it will attempt to
> cleanly destroy any consumers before it is terminated."
>
> Does the highlighted mean that I have to have multiple deployed
> instances in order to have more than 1 consumer group and configure it
> with sticky sessions?
>
You can run multiple consumer instances in one or more consumer groups on a single REST proxy instance. However, you'll need to ensure they have different IDs (which you can have auto-assigned if you like).
>
> I have the following code where I am testing the behavior when I have
> 2 consumers in the same group, and I am getting an empty array back
> for the second consumer in the same group.
>
> This will through an exception when I attempt to read a message from
> the 2nd consumer.
>
> Not sure if it would help, but I configured the proxy to have 2
> threads per consumer.
>
> KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("
> http://10.1.30.48:8082");
>
>
>
> KafkaMessageQueueReader reader1 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader2 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader3 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48"
> });
>
>
>
> for (int i = 0; i < 10; i++)
> {
> //Publish a message to the ShellComparisonTest topic
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> var @event =
> reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> @event =
> reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> //Publish a message to the ShellComparisonTest topic
> so that
> //there is one unread message in the queue before trying to
> //read from the second consumer in the group
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> @event =
> reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
> }
>
> Is there something that I am missing or is there a good resource to read.
>
>
How many partitions does your topic have? Did you create it before this test or was it auto-created? One reason you could see this behavior is if you only have 1 partition. In that case only 1 consumer would see any messages since only 1 consumer would have any partitions assigned to it.
-Ewen
> Thanks
>
> Heath
>
>
> Warning: This e-mail may contain information proprietary to
> AutoAnything Inc. and is intended only for the use of the intended
> recipient(s). If the reader of this message is not the intended
> recipient(s), you have received this message in error and any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. If you have received this message in error, please notify
> the sender immediately and delete all copies.
>
--
Thanks,
Ewen
RE: Kafka-Rest question
Posted by Heath Ivie <hi...@AutoAnything.com>.
Thanks Ewen.
I have that in my example, but when there are two consumers each with an autogenerated id, only one can read and get data.
The second one will return an empty array, even with a message being published after the first consumer is finished.
I see what is expected if I use the .NET client, but in the proxy it doesn’t appear to work like that.
-----Original Message-----
From: Ewen Cheslack-Postava [mailto:ewen@confluent.io]
Sent: Thursday, January 07, 2016 1:18 PM
To: users@kafka.apache.org
Subject: Re: Kafka-Rest question
On Thu, Jan 7, 2016 at 12:54 PM, Heath Ivie <hi...@autoanything.com> wrote:
> Hi,
>
> I am building my application using the rest proxy and I wanted to get
> some clarification on the documentation.
>
> "Because consumers are stateful, any consumer instances created with
> the REST API are tied to a specific REST proxy instance. A full URL is
> provided when the instance is created and it should be used to
> construct any subsequent requests. Failing to use the returned URL for
> future consumer requests will end up adding new consumers to the
> group. If a REST proxy instance is shutdown, it will attempt to
> cleanly destroy any consumers before it is terminated."
>
> Does the highlighted mean that I have to have multiple deployed
> instances in order to have more than 1 consumer group and configure it
> with sticky sessions?
>
You can run multiple consumer instances in one or more consumer groups on a single REST proxy instance. However, you'll need to ensure they have different IDs (which you can have auto-assigned if you like).
>
> I have the following code where I am testing the behavior when I have
> 2 consumers in the same group, and I am getting an empty array back
> for the second consumer in the same group.
>
> This will through an exception when I attempt to read a message from
> the 2nd consumer.
>
> Not sure if it would help, but I configured the proxy to have 2
> threads per consumer.
>
> KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("
> http://10.1.30.48:8082");
>
>
>
> KafkaMessageQueueReader reader1 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader2 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader3 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48"
> });
>
>
>
> for (int i = 0; i < 10; i++)
> {
> //Publish a message to the ShellComparisonTest topic
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> var @event =
> reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> @event =
> reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> //Publish a message to the ShellComparisonTest topic
> so that
> //there is one unread message in the queue before trying to
> //read from the second consumer in the group
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> @event =
> reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
> }
>
> Is there something that I am missing or is there a good resource to read.
>
>
How many partitions does your topic have? Did you create it before this test or was it auto-created? One reason you could see this behavior is if you only have 1 partition. In that case only 1 consumer would see any messages since only 1 consumer would have any partitions assigned to it.
-Ewen
> Thanks
>
> Heath
>
>
> Warning: This e-mail may contain information proprietary to
> AutoAnything Inc. and is intended only for the use of the intended
> recipient(s). If the reader of this message is not the intended
> recipient(s), you have received this message in error and any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. If you have received this message in error, please notify
> the sender immediately and delete all copies.
>
--
Thanks,
Ewen
Re: Kafka-Rest question
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
On Thu, Jan 7, 2016 at 12:54 PM, Heath Ivie <hi...@autoanything.com> wrote:
> Hi,
>
> I am building my application using the rest proxy and I wanted to get some
> clarification on the documentation.
>
> "Because consumers are stateful, any consumer instances created with the
> REST API are tied to a specific REST proxy instance. A full URL is provided
> when the instance is created and it should be used to construct any
> subsequent requests. Failing to use the returned URL for future consumer
> requests will end up adding new consumers to the group. If a REST proxy
> instance is shutdown, it will attempt to cleanly destroy any consumers
> before it is terminated."
>
> Does the highlighted mean that I have to have multiple deployed instances
> in order to have more than 1 consumer group and configure it with sticky
> sessions?
>
You can run multiple consumer instances in one or more consumer groups on a
single REST proxy instance. However, you'll need to ensure they have
different IDs (which you can have auto-assigned if you like).
>
> I have the following code where I am testing the behavior when I have 2
> consumers in the same group, and I am getting an empty array back for the
> second consumer in the same group.
>
> This will through an exception when I attempt to read a message from the
> 2nd consumer.
>
> Not sure if it would help, but I configured the proxy to have 2 threads
> per consumer.
>
> KafkaMessageQueueWriter writer = new KafkaMessageQueueWriter("
> http://10.1.30.48:8082");
>
>
>
> KafkaMessageQueueReader reader1 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader2 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test51", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
> KafkaMessageQueueReader reader3 = new
> KafkaMessageQueueReader(new KafkaMessageQueueReaderSettings() {
> ConsumerGroupName = "Test50", TopicName = "ShellComparisonTest",
> ZookeeperClientPort = 8082, ZookeeperClientPortAddress = "10.1.30.48" });
>
>
>
> for (int i = 0; i < 10; i++)
> {
> //Publish a message to the ShellComparisonTest topic
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> var @event =
> reader1.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> @event =
> reader2.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
>
> //Publish a message to the ShellComparisonTest topic so
> that
> //there is one unread message in the queue before trying to
> //read from the second consumer in the group
> writer.PublishMessage("ShellComparisonTest", new
> OrderEvent() { OrderId = 5, OrderItemId = 1234 });
> @event =
> reader3.ReadQueue<OrderEvent>("ShellComparisonTest");
> if (!@event.Any()) throw new Exception();
> }
>
> Is there something that I am missing or is there a good resource to read.
>
>
How many partitions does your topic have? Did you create it before this
test or was it auto-created? One reason you could see this behavior is if
you only have 1 partition. In that case only 1 consumer would see any
messages since only 1 consumer would have any partitions assigned to it.
-Ewen
> Thanks
>
> Heath
>
>
> Warning: This e-mail may contain information proprietary to AutoAnything
> Inc. and is intended only for the use of the intended recipient(s). If the
> reader of this message is not the intended recipient(s), you have received
> this message in error and any review, dissemination, distribution or
> copying of this message is strictly prohibited. If you have received this
> message in error, please notify the sender immediately and delete all
> copies.
>
--
Thanks,
Ewen