You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Srinivas Reddy Kancharla <ge...@gmail.com> on 2014/11/08 00:34:46 UTC
Consumer thread dies
Hi,
I have a scenario where I have 1 partition and 1 consumer group having 2
consumer threads running say C1 and C2. Since there is only one partition
for a given topic, say C1 is holding that partition. Now due to some reason
if C1 dies, can C2 get hold of that partition?
i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in
hung state, Can we make C2 talking to KafkaStream (for Partition 0).
I am facing this issue where I have 10 messages in partition 0 and C1 was
consuming it. At message 4, C1 went into hung state. Now I would like to
make C2 to consumer other messages which are not consumed by C1.
Thank and regards,
Srini
Re: Consumer thread dies
Posted by Srinivas Reddy Kancharla <ge...@gmail.com>.
Hi,
Further I looked at this scenario, Is it correct that above scenario can be
handled if I use SimpleConsumer approach instead of using
"ConsumerConnector.createMessageStreams()" , this way I have better control
on partition. This way my partition is not bound with any specific consumer
thread. Please let me know if I am missing anything with
"ConsumerConnector.createMessageStreams()".
Thanks and regards,
Srini
On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla <getreddy@gmail.com
> wrote:
> Hi,
>
> I have a scenario where I have 1 partition and 1 consumer group having 2
> consumer threads running say C1 and C2. Since there is only one partition
> for a given topic, say C1 is holding that partition. Now due to some reason
> if C1 dies, can C2 get hold of that partition?
>
> i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or
> in hung state, Can we make C2 talking to KafkaStream (for Partition 0).
> I am facing this issue where I have 10 messages in partition 0 and C1 was
> consuming it. At message 4, C1 went into hung state. Now I would like to
> make C2 to consumer other messages which are not consumed by C1.
>
> Thank and regards,
> Srini
>
Re: Consumer thread dies
Posted by Jun Rao <ju...@gmail.com>.
Don't you have the same problem using SimpleConsumer? How does another
process know a SimpleConsumer hangs?
Thanks,
Jun
On Mon, Nov 10, 2014 at 9:47 AM, Srinivas Reddy Kancharla <
getreddy@gmail.com> wrote:
> Thanks Jun for your response.
> Here is my scenario:
>
> topicCountMap.put(topic, new Integer(2));
> Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
> consumer.createMessageStreams(topicCountMap);
> List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
>
> So from above scenario (only 1 partition) , there will be 2 threads C1 and
> C2, and one of the thread would be holding one partition. In my scenario,
> C1 would be hung after consuming the message from stream and in hung state
> during processing of message. So in such scenario, looks like C2 will not
> get hold of partition stream unless C1 didn't die..
>
> So considering such scenarios, I am planning to use SimpleConsumer where I
> will have more control on consuming Partition content. But I have to
> maintain offset state in zookeeper. Let me know if this approach is correct
> for such hung scenarios.
>
> Thanks and regards,
> Srini
>
>
> On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > If C1 dies, C2 will be owning that partition. However, C1 has to really
> > die, which typically means that either you close the consumer connector
> or
> > the jvm of C1 is gone.
> >
> > In your case, it seems that C1 didn't die, it just hung. Do you know why
> C1
> > hung?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla <
> > getreddy@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > > I have a scenario where I have 1 partition and 1 consumer group having
> 2
> > > consumer threads running say C1 and C2. Since there is only one
> partition
> > > for a given topic, say C1 is holding that partition. Now due to some
> > reason
> > > if C1 dies, can C2 get hold of that partition?
> > >
> > > i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies
> or
> > in
> > > hung state, Can we make C2 talking to KafkaStream (for Partition 0).
> > > I am facing this issue where I have 10 messages in partition 0 and C1
> was
> > > consuming it. At message 4, C1 went into hung state. Now I would like
> to
> > > make C2 to consumer other messages which are not consumed by C1.
> > >
> > > Thank and regards,
> > > Srini
> > >
> >
>
Re: Consumer thread dies
Posted by Srinivas Reddy Kancharla <ge...@gmail.com>.
Thanks Jun for your response.
Here is my scenario:
topicCountMap.put(topic, new Integer(2));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
So from above scenario (only 1 partition) , there will be 2 threads C1 and
C2, and one of the thread would be holding one partition. In my scenario,
C1 would be hung after consuming the message from stream and in hung state
during processing of message. So in such scenario, looks like C2 will not
get hold of partition stream unless C1 didn't die..
So considering such scenarios, I am planning to use SimpleConsumer where I
will have more control on consuming Partition content. But I have to
maintain offset state in zookeeper. Let me know if this approach is correct
for such hung scenarios.
Thanks and regards,
Srini
On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao <ju...@gmail.com> wrote:
> If C1 dies, C2 will be owning that partition. However, C1 has to really
> die, which typically means that either you close the consumer connector or
> the jvm of C1 is gone.
>
> In your case, it seems that C1 didn't die, it just hung. Do you know why C1
> hung?
>
> Thanks,
>
> Jun
>
> On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla <
> getreddy@gmail.com
> > wrote:
>
> > Hi,
> >
> > I have a scenario where I have 1 partition and 1 consumer group having 2
> > consumer threads running say C1 and C2. Since there is only one partition
> > for a given topic, say C1 is holding that partition. Now due to some
> reason
> > if C1 dies, can C2 get hold of that partition?
> >
> > i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or
> in
> > hung state, Can we make C2 talking to KafkaStream (for Partition 0).
> > I am facing this issue where I have 10 messages in partition 0 and C1 was
> > consuming it. At message 4, C1 went into hung state. Now I would like to
> > make C2 to consumer other messages which are not consumed by C1.
> >
> > Thank and regards,
> > Srini
> >
>
Re: Consumer thread dies
Posted by Jun Rao <ju...@gmail.com>.
If C1 dies, C2 will be owning that partition. However, C1 has to really
die, which typically means that either you close the consumer connector or
the jvm of C1 is gone.
In your case, it seems that C1 didn't die, it just hung. Do you know why C1
hung?
Thanks,
Jun
On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla <getreddy@gmail.com
> wrote:
> Hi,
>
> I have a scenario where I have 1 partition and 1 consumer group having 2
> consumer threads running say C1 and C2. Since there is only one partition
> for a given topic, say C1 is holding that partition. Now due to some reason
> if C1 dies, can C2 get hold of that partition?
>
> i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies or in
> hung state, Can we make C2 talking to KafkaStream (for Partition 0).
> I am facing this issue where I have 10 messages in partition 0 and C1 was
> consuming it. At message 4, C1 went into hung state. Now I would like to
> make C2 to consumer other messages which are not consumed by C1.
>
> Thank and regards,
> Srini
>