You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 4mayank <4m...@gmail.com> on 2015/04/13 18:55:41 UTC
Re: Consumer offsets in offsets topic 0.8.2
I did a similar change - moved from High Level Consumer to Simple Consumer.
Howerver kafka-consumer-offset-checker.sh throws an exception. Its
searching the zk path /consumers/<group>/ which does not exist on any of my
zk nodes.
Is there any other tool for getting the offset lag when using Simple
Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for
Simple Consumer?
Output:
kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
192.168.1.202:2181,192.168.1.203:2181 --group my-control-group
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/my-control-group/owners.
kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic
my-control
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/consumers/my-control-group/offsets/my-control/1.
ZK cli output:
./zookeeper-shell.sh 192.168.1.201:2181
Connecting to 192.168.1.201:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /config/topics
[my-control]
ls /consumers
[]
quit
Thanks.
Mayank.
On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> Hi Vamsi,
>
> The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh still
> works. You can use them to check the offsets.
> If you need to check the offsets programmatically, you can send
> OffsetsFetcheRequest to broker using simple consumer. You may refer the
> ConsumerOffsetChecker.scala to see how to find correct broker where
> corresponding offset manager resides.
>
> Jiangjie (Becket) Qin
>
> On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
> wrote:
>
> >Hi,
> >
> >We are using 0.8.2.1 currently.
> >
> >- How to get the consumer offsets from the offsets topic?
> >- Is there any built-in function which I could use? (like in
> >AdminUtils.scala)
> >- Is it ok to start a simple consumer and read the offsets from the topic?
> >
> >We used to read the offsets from zookeeper previously for the
> >HighLevelConsumers. But with the new broker this changed as we are using
> >Kafka topic for offsets.
> >
> >--
> >Regards
> >Vamsi Subhash
>
>
Re: Consumer offsets in offsets topic 0.8.2
Posted by Madhukar Bharti <bh...@gmail.com>.
Hi Vamsi,
You can also see the example here
<https://github.com/madhukarbharti/kafka-8.2.1-test/blob/master/src/com/bharti/kafka/offset/OffsetHandler.java>
if
you want to use Java API to get the offset from topic.
Regards,
Madhukar
On Mon, Apr 13, 2015 at 10:25 PM, 4mayank <4m...@gmail.com> wrote:
> I did a similar change - moved from High Level Consumer to Simple Consumer.
> Howerver kafka-consumer-offset-checker.sh throws an exception. Its
> searching the zk path /consumers/<group>/ which does not exist on any of my
> zk nodes.
>
> Is there any other tool for getting the offset lag when using Simple
> Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for
> Simple Consumer?
>
> Output:
> kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
> 192.168.1.202:2181,192.168.1.203:2181 --group my-control-group
> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /consumers/my-control-group/owners.
>
>
> kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
> 192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic
> my-control
> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /consumers/my-control-group/offsets/my-control/1.
>
>
> ZK cli output:
>
> ./zookeeper-shell.sh 192.168.1.201:2181
> Connecting to 192.168.1.201:2181
> Welcome to ZooKeeper!
> JLine support is disabled
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> ls /config/topics
> [my-control]
> ls /consumers
> []
> quit
>
>
> Thanks.
> Mayank.
>
> On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > Hi Vamsi,
> >
> > The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh still
> > works. You can use them to check the offsets.
> > If you need to check the offsets programmatically, you can send
> > OffsetsFetcheRequest to broker using simple consumer. You may refer the
> > ConsumerOffsetChecker.scala to see how to find correct broker where
> > corresponding offset manager resides.
> >
> > Jiangjie (Becket) Qin
> >
> > On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash" <
> achanta.vamsi@flipkart.com>
> > wrote:
> >
> > >Hi,
> > >
> > >We are using 0.8.2.1 currently.
> > >
> > >- How to get the consumer offsets from the offsets topic?
> > >- Is there any built-in function which I could use? (like in
> > >AdminUtils.scala)
> > >- Is it ok to start a simple consumer and read the offsets from the
> topic?
> > >
> > >We used to read the offsets from zookeeper previously for the
> > >HighLevelConsumers. But with the new broker this changed as we are using
> > >Kafka topic for offsets.
> > >
> > >--
> > >Regards
> > >Vamsi Subhash
> >
> >
>
Re: Consumer offsets in offsets topic 0.8.2
Posted by 4mayank <4m...@gmail.com>.
Thanks guys. Based on the responses it looks like I need to have a tool of
my own to read offsets.
On Mon, Apr 13, 2015 at 3:07 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> Yeah, the current ConsumerOffsetChecker has this issue (maybe bug also) if
> the offset storage is Kafka and no offset has been committed. It will
> throw ZK exception, which is very confusing. KAFKA-1951 was opened for
> this but was not checked in.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>
> On 4/13/15, 9:55 AM, "4mayank" <4m...@gmail.com> wrote:
>
> >I did a similar change - moved from High Level Consumer to Simple
> >Consumer.
> >Howerver kafka-consumer-offset-checker.sh throws an exception. Its
> >searching the zk path /consumers/<group>/ which does not exist on any of
> >my
> >zk nodes.
> >
> >Is there any other tool for getting the offset lag when using Simple
> >Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for
> >Simple Consumer?
> >
> >Output:
> >kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
> >192.168.1.202:2181,192.168.1.203:2181 --group my-control-group
> >Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> >KeeperErrorCode = NoNode for /consumers/my-control-group/owners.
> >
> >
> >kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
> >192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic
> >my-control
> >Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> >KeeperErrorCode = NoNode for
> >/consumers/my-control-group/offsets/my-control/1.
> >
> >
> >ZK cli output:
> >
> >./zookeeper-shell.sh 192.168.1.201:2181
> >Connecting to 192.168.1.201:2181
> >Welcome to ZooKeeper!
> >JLine support is disabled
> >
> >WATCHER::
> >
> >WatchedEvent state:SyncConnected type:None path:null
> >ls /config/topics
> >[my-control]
> >ls /consumers
> >[]
> >quit
> >
> >
> >Thanks.
> >Mayank.
> >
> >On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> >wrote:
> >
> >> Hi Vamsi,
> >>
> >> The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh
> >>still
> >> works. You can use them to check the offsets.
> >> If you need to check the offsets programmatically, you can send
> >> OffsetsFetcheRequest to broker using simple consumer. You may refer the
> >> ConsumerOffsetChecker.scala to see how to find correct broker where
> >> corresponding offset manager resides.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash"
> >><ac...@flipkart.com>
> >> wrote:
> >>
> >> >Hi,
> >> >
> >> >We are using 0.8.2.1 currently.
> >> >
> >> >- How to get the consumer offsets from the offsets topic?
> >> >- Is there any built-in function which I could use? (like in
> >> >AdminUtils.scala)
> >> >- Is it ok to start a simple consumer and read the offsets from the
> >>topic?
> >> >
> >> >We used to read the offsets from zookeeper previously for the
> >> >HighLevelConsumers. But with the new broker this changed as we are
> >>using
> >> >Kafka topic for offsets.
> >> >
> >> >--
> >> >Regards
> >> >Vamsi Subhash
> >>
> >>
>
>
Re: Consumer offsets in offsets topic 0.8.2
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Yeah, the current ConsumerOffsetChecker has this issue (maybe bug also) if
the offset storage is Kafka and no offset has been committed. It will
throw ZK exception, which is very confusing. KAFKA-1951 was opened for
this but was not checked in.
Thanks.
Jiangjie (Becket) Qin
On 4/13/15, 9:55 AM, "4mayank" <4m...@gmail.com> wrote:
>I did a similar change - moved from High Level Consumer to Simple
>Consumer.
>Howerver kafka-consumer-offset-checker.sh throws an exception. Its
>searching the zk path /consumers/<group>/ which does not exist on any of
>my
>zk nodes.
>
>Is there any other tool for getting the offset lag when using Simple
>Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for
>Simple Consumer?
>
>Output:
>kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
>192.168.1.202:2181,192.168.1.203:2181 --group my-control-group
>Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>KeeperErrorCode = NoNode for /consumers/my-control-group/owners.
>
>
>kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181,
>192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic
>my-control
>Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
>KeeperErrorCode = NoNode for
>/consumers/my-control-group/offsets/my-control/1.
>
>
>ZK cli output:
>
>./zookeeper-shell.sh 192.168.1.201:2181
>Connecting to 192.168.1.201:2181
>Welcome to ZooKeeper!
>JLine support is disabled
>
>WATCHER::
>
>WatchedEvent state:SyncConnected type:None path:null
>ls /config/topics
>[my-control]
>ls /consumers
>[]
>quit
>
>
>Thanks.
>Mayank.
>
>On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
>wrote:
>
>> Hi Vamsi,
>>
>> The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh
>>still
>> works. You can use them to check the offsets.
>> If you need to check the offsets programmatically, you can send
>> OffsetsFetcheRequest to broker using simple consumer. You may refer the
>> ConsumerOffsetChecker.scala to see how to find correct broker where
>> corresponding offset manager resides.
>>
>> Jiangjie (Becket) Qin
>>
>> On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash"
>><ac...@flipkart.com>
>> wrote:
>>
>> >Hi,
>> >
>> >We are using 0.8.2.1 currently.
>> >
>> >- How to get the consumer offsets from the offsets topic?
>> >- Is there any built-in function which I could use? (like in
>> >AdminUtils.scala)
>> >- Is it ok to start a simple consumer and read the offsets from the
>>topic?
>> >
>> >We used to read the offsets from zookeeper previously for the
>> >HighLevelConsumers. But with the new broker this changed as we are
>>using
>> >Kafka topic for offsets.
>> >
>> >--
>> >Regards
>> >Vamsi Subhash
>>
>>