You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gürkan Oluç <gu...@gmail.com> on 2014/01/15 19:58:19 UTC
Consumer starts to consume after changing group id
Hello,
We are using Kafka 0.8 with Storm. To consume from Kafka, we are using
https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We are
creating spout with this configuration :
https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9
Our problem is, consumers stops after working some time. From storm logs, I
am seeing that consumer offset is increasing. But, Storm is not emitting
any thing to other bolts.
Also, we had almost same issue with another classical java application. It
stops after some time, but when I change the group id of consumer group or
restart process it's starting to consume again. Changing group id sometimes
work for storm spout. My consumer configuration for this classic app is
https://gist.github.com/gurkanoluc/da46e7df263cfaffca16
I have looked to
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3Fthis
topic from topic but, It doesn't help me. Is there anything to debug
this issue?
Instead of consuming with storm, If I consume with kafka's own scripts to
consume. It consumes successfully.
Thanks for your help.
Best,
--
Gürkan OLUÇ
http://www.gurkanoluc.com
Re: Consumer starts to consume after changing group id
Posted by Gürkan Oluç <gu...@gmail.com>.
Hello Jun,
Thanks for your answer. I agree with you, I think problem is Storm 0.9
Spout.
Thanks,
Gurkan
On Thu, Jan 16, 2014 at 5:49 PM, Jun Rao <ju...@gmail.com> wrote:
> If the Kafka consumption part is working, you would have to look into Storm
> to see why the messages are dropped.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jan 15, 2014 at 11:32 PM, Gürkan Oluç <gu...@gmail.com>
> wrote:
>
> > Hello,
> >
> > It looks like it consumes from Kafka. here is the request log :
> > https://gist.github.com/gurkanoluc/39d2762ade4e2148a20c
> >
> >
> >
> >
> > On Thu, Jan 16, 2014 at 7:29 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > It seems that it's using SimpleConsumer. Could you check Kafka broker's
> > > request log and see if the consumer is making the right fetch request?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Jan 15, 2014 at 10:58 AM, Gürkan Oluç <gu...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > We are using Kafka 0.8 with Storm. To consume from Kafka, we are
> using
> > > > https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We
> > are
> > > > creating spout with this configuration :
> > > > https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9
> > > >
> > > > Our problem is, consumers stops after working some time. From storm
> > > logs, I
> > > > am seeing that consumer offset is increasing. But, Storm is not
> > emitting
> > > > any thing to other bolts.
> > > >
> > > > Also, we had almost same issue with another classical java
> application.
> > > It
> > > > stops after some time, but when I change the group id of consumer
> group
> > > or
> > > > restart process it's starting to consume again. Changing group id
> > > sometimes
> > > > work for storm spout. My consumer configuration for this classic app
> is
> > > > https://gist.github.com/gurkanoluc/da46e7df263cfaffca16
> > > >
> > > > I have looked to
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3Fthis
> > > > topic from topic but, It doesn't help me. Is there anything to debug
> > > > this issue?
> > > >
> > > > Instead of consuming with storm, If I consume with kafka's own
> scripts
> > to
> > > > consume. It consumes successfully.
> > > >
> > > > Thanks for your help.
> > > > Best,
> > > >
> > > > --
> > > > Gürkan OLUÇ
> > > > http://www.gurkanoluc.com
> > > >
> > >
> >
> >
> >
> > --
> > Gürkan OLUÇ
> > http://www.gurkanoluc.com
> >
>
--
Gürkan OLUÇ
http://www.gurkanoluc.com
Re: Consumer starts to consume after changing group id
Posted by Jun Rao <ju...@gmail.com>.
If the Kafka consumption part is working, you would have to look into Storm
to see why the messages are dropped.
Thanks,
Jun
On Wed, Jan 15, 2014 at 11:32 PM, Gürkan Oluç <gu...@gmail.com> wrote:
> Hello,
>
> It looks like it consumes from Kafka. here is the request log :
> https://gist.github.com/gurkanoluc/39d2762ade4e2148a20c
>
>
>
>
> On Thu, Jan 16, 2014 at 7:29 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > It seems that it's using SimpleConsumer. Could you check Kafka broker's
> > request log and see if the consumer is making the right fetch request?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Jan 15, 2014 at 10:58 AM, Gürkan Oluç <gu...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We are using Kafka 0.8 with Storm. To consume from Kafka, we are using
> > > https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We
> are
> > > creating spout with this configuration :
> > > https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9
> > >
> > > Our problem is, consumers stops after working some time. From storm
> > logs, I
> > > am seeing that consumer offset is increasing. But, Storm is not
> emitting
> > > any thing to other bolts.
> > >
> > > Also, we had almost same issue with another classical java application.
> > It
> > > stops after some time, but when I change the group id of consumer group
> > or
> > > restart process it's starting to consume again. Changing group id
> > sometimes
> > > work for storm spout. My consumer configuration for this classic app is
> > > https://gist.github.com/gurkanoluc/da46e7df263cfaffca16
> > >
> > > I have looked to
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3Fthis
> > > topic from topic but, It doesn't help me. Is there anything to debug
> > > this issue?
> > >
> > > Instead of consuming with storm, If I consume with kafka's own scripts
> to
> > > consume. It consumes successfully.
> > >
> > > Thanks for your help.
> > > Best,
> > >
> > > --
> > > Gürkan OLUÇ
> > > http://www.gurkanoluc.com
> > >
> >
>
>
>
> --
> Gürkan OLUÇ
> http://www.gurkanoluc.com
>
Re: Consumer starts to consume after changing group id
Posted by Gürkan Oluç <gu...@gmail.com>.
Hello,
It looks like it consumes from Kafka. here is the request log :
https://gist.github.com/gurkanoluc/39d2762ade4e2148a20c
On Thu, Jan 16, 2014 at 7:29 AM, Jun Rao <ju...@gmail.com> wrote:
> It seems that it's using SimpleConsumer. Could you check Kafka broker's
> request log and see if the consumer is making the right fetch request?
>
> Thanks,
>
> Jun
>
>
> On Wed, Jan 15, 2014 at 10:58 AM, Gürkan Oluç <gu...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We are using Kafka 0.8 with Storm. To consume from Kafka, we are using
> > https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We are
> > creating spout with this configuration :
> > https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9
> >
> > Our problem is, consumers stops after working some time. From storm
> logs, I
> > am seeing that consumer offset is increasing. But, Storm is not emitting
> > any thing to other bolts.
> >
> > Also, we had almost same issue with another classical java application.
> It
> > stops after some time, but when I change the group id of consumer group
> or
> > restart process it's starting to consume again. Changing group id
> sometimes
> > work for storm spout. My consumer configuration for this classic app is
> > https://gist.github.com/gurkanoluc/da46e7df263cfaffca16
> >
> > I have looked to
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3Fthis
> > topic from topic but, It doesn't help me. Is there anything to debug
> > this issue?
> >
> > Instead of consuming with storm, If I consume with kafka's own scripts to
> > consume. It consumes successfully.
> >
> > Thanks for your help.
> > Best,
> >
> > --
> > Gürkan OLUÇ
> > http://www.gurkanoluc.com
> >
>
--
Gürkan OLUÇ
http://www.gurkanoluc.com
Re: Consumer starts to consume after changing group id
Posted by Jun Rao <ju...@gmail.com>.
It seems that it's using SimpleConsumer. Could you check Kafka broker's
request log and see if the consumer is making the right fetch request?
Thanks,
Jun
On Wed, Jan 15, 2014 at 10:58 AM, Gürkan Oluç <gu...@gmail.com> wrote:
> Hello,
>
> We are using Kafka 0.8 with Storm. To consume from Kafka, we are using
> https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We are
> creating spout with this configuration :
> https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9
>
> Our problem is, consumers stops after working some time. From storm logs, I
> am seeing that consumer offset is increasing. But, Storm is not emitting
> any thing to other bolts.
>
> Also, we had almost same issue with another classical java application. It
> stops after some time, but when I change the group id of consumer group or
> restart process it's starting to consume again. Changing group id sometimes
> work for storm spout. My consumer configuration for this classic app is
> https://gist.github.com/gurkanoluc/da46e7df263cfaffca16
>
> I have looked to
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3Fthis
> topic from topic but, It doesn't help me. Is there anything to debug
> this issue?
>
> Instead of consuming with storm, If I consume with kafka's own scripts to
> consume. It consumes successfully.
>
> Thanks for your help.
> Best,
>
> --
> Gürkan OLUÇ
> http://www.gurkanoluc.com
>