You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Amoxicillin <sh...@gmail.com> on 2016/02/19 05:12:45 UTC
ConsumerGroupCommand and ConsumerOffsetChecker
Hi,
I use kafka.tools.ConsumerOffsetChecker to view the consumer offset
status, and can get the correct output, along with a
warning: ConsumerOffsetChecker is deprecated and will be dropped in
releases following 0.9.0. Use ConsumerGroupCommand instead.
But when I altered to bin/kafka-run-class.sh
kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181 --list, there
was nothing to output.
Did I missed some parameters to run ConsumerGroupCommand? The version is
kafka _2.11-0.9.0.0, and any suggestion is appreciated.
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by tao xiao <xi...@gmail.com>.
I am not familiar with sarama. You can use zkcli.sh from zookeeper
distribution and find if anything exists in path /chroot/consumers where
chroot is located in your bootstrap.servers properties if you set it. If
you dont find anything in /consumers offset is most likely managed in
kafka. Then you should be able to find the groups in kafka-run-class.sh
kafka.admin.ConsumerGroupCommand --list --new-consumer --bootstrap-server
localhost:9092. You should try it for a couple of times sometimes the
command has delay to show info.
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
On Fri, 19 Feb 2016 at 16:48 Amoxicillin <sh...@gmail.com> wrote:
> I suppose the offset is stored in kafka not in zk, because the newest
> sarama's offset manager support it, and it shown a topic named
> __consumer_offsets.
> Could you please point where is the zk path of /chroot/consumer? I'm a
> newbie of zk and kafka.
>
> On Fri, Feb 19, 2016 at 4:11 PM, tao xiao <xi...@gmail.com> wrote:
>
> > I'd assume the offset is managed in zk. then kafka-run-class.sh
> > kafka.admin.ConsumerGroupCommand --list --zookeeper localhost:2181 should
> > output the consumer group that is saved in zk. can you check what is
> stored
> > in zk path /chroot/consumers?
> >
> > On Fri, 19 Feb 2016 at 16:06 Amoxicillin <sh...@gmail.com> wrote:
> >
> > > I wrote the consumer by myself in golang using Shopify's sarama lib
> > updated
> > > of master branch.
> > >
> > > On Fri, Feb 19, 2016 at 4:03 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > which version of consumer do you use?
> > > >
> > > >
> > > > On Fri, 19 Feb 2016 at 15:26 Amoxicillin <sh...@gmail.com> wrote:
> > > >
> > > > > I tried as you suggested, but still no output of any group info.
> > > > >
> > > > > On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > That is what I mean alive. If you use new consumer connecting to
> > > > broker
> > > > > > you should use --new-consumer option to list all consumer groups
> > > > > >
> > > > > > kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list
> > > > --new-consumer
> > > > > > --bootstrap-server localhost:9092
> > > > > >
> > > > > > On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > How to confirm the consumer groups are alive? I have one
> consumer
> > > in
> > > > > the
> > > > > > > group running at the same time, and could get messages
> correctly.
> > > > > > >
> > > > > > > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <
> xiaotao183@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > when using ConsumerGroupCommand you need to make sure your
> > > consumer
> > > > > > > groups
> > > > > > > > are alive. It only queries offsets for consumer groups that
> are
> > > > > > currently
> > > > > > > > connecting to brokers
> > > > > > > >
> > > > > > > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I use kafka.tools.ConsumerOffsetChecker to view the
> consumer
> > > > > offset
> > > > > > > > > status, and can get the correct output, along with a
> > > > > > > > > warning: ConsumerOffsetChecker is deprecated and will be
> > > dropped
> > > > in
> > > > > > > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > > > > > > >
> > > > > > > > > But when I altered to bin/kafka-run-class.sh
> > > > > > > > > kafka.admin.ConsumerGroupCommand --zookeeper
> 127.0.0.1:2181
> > > > > --list,
> > > > > > > > there
> > > > > > > > > was nothing to output.
> > > > > > > > >
> > > > > > > > > Did I missed some parameters to run ConsumerGroupCommand?
> The
> > > > > version
> > > > > > > is
> > > > > > > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by Amoxicillin <sh...@gmail.com>.
I suppose the offset is stored in kafka not in zk, because the newest
sarama's offset manager support it, and it shown a topic named
__consumer_offsets.
Could you please point where is the zk path of /chroot/consumer? I'm a
newbie of zk and kafka.
On Fri, Feb 19, 2016 at 4:11 PM, tao xiao <xi...@gmail.com> wrote:
> I'd assume the offset is managed in zk. then kafka-run-class.sh
> kafka.admin.ConsumerGroupCommand --list --zookeeper localhost:2181 should
> output the consumer group that is saved in zk. can you check what is stored
> in zk path /chroot/consumers?
>
> On Fri, 19 Feb 2016 at 16:06 Amoxicillin <sh...@gmail.com> wrote:
>
> > I wrote the consumer by myself in golang using Shopify's sarama lib
> updated
> > of master branch.
> >
> > On Fri, Feb 19, 2016 at 4:03 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > which version of consumer do you use?
> > >
> > >
> > > On Fri, 19 Feb 2016 at 15:26 Amoxicillin <sh...@gmail.com> wrote:
> > >
> > > > I tried as you suggested, but still no output of any group info.
> > > >
> > > > On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > That is what I mean alive. If you use new consumer connecting to
> > > broker
> > > > > you should use --new-consumer option to list all consumer groups
> > > > >
> > > > > kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list
> > > --new-consumer
> > > > > --bootstrap-server localhost:9092
> > > > >
> > > > > On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > How to confirm the consumer groups are alive? I have one consumer
> > in
> > > > the
> > > > > > group running at the same time, and could get messages correctly.
> > > > > >
> > > > > > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > when using ConsumerGroupCommand you need to make sure your
> > consumer
> > > > > > groups
> > > > > > > are alive. It only queries offsets for consumer groups that are
> > > > > currently
> > > > > > > connecting to brokers
> > > > > > >
> > > > > > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I use kafka.tools.ConsumerOffsetChecker to view the consumer
> > > > offset
> > > > > > > > status, and can get the correct output, along with a
> > > > > > > > warning: ConsumerOffsetChecker is deprecated and will be
> > dropped
> > > in
> > > > > > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > > > > > >
> > > > > > > > But when I altered to bin/kafka-run-class.sh
> > > > > > > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181
> > > > --list,
> > > > > > > there
> > > > > > > > was nothing to output.
> > > > > > > >
> > > > > > > > Did I missed some parameters to run ConsumerGroupCommand? The
> > > > version
> > > > > > is
> > > > > > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by tao xiao <xi...@gmail.com>.
I'd assume the offset is managed in zk. then kafka-run-class.sh
kafka.admin.ConsumerGroupCommand --list --zookeeper localhost:2181 should
output the consumer group that is saved in zk. can you check what is stored
in zk path /chroot/consumers?
On Fri, 19 Feb 2016 at 16:06 Amoxicillin <sh...@gmail.com> wrote:
> I wrote the consumer by myself in golang using Shopify's sarama lib updated
> of master branch.
>
> On Fri, Feb 19, 2016 at 4:03 PM, tao xiao <xi...@gmail.com> wrote:
>
> > which version of consumer do you use?
> >
> >
> > On Fri, 19 Feb 2016 at 15:26 Amoxicillin <sh...@gmail.com> wrote:
> >
> > > I tried as you suggested, but still no output of any group info.
> > >
> > > On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > That is what I mean alive. If you use new consumer connecting to
> > broker
> > > > you should use --new-consumer option to list all consumer groups
> > > >
> > > > kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list
> > --new-consumer
> > > > --bootstrap-server localhost:9092
> > > >
> > > > On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com> wrote:
> > > >
> > > > > How to confirm the consumer groups are alive? I have one consumer
> in
> > > the
> > > > > group running at the same time, and could get messages correctly.
> > > > >
> > > > > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > when using ConsumerGroupCommand you need to make sure your
> consumer
> > > > > groups
> > > > > > are alive. It only queries offsets for consumer groups that are
> > > > currently
> > > > > > connecting to brokers
> > > > > >
> > > > > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I use kafka.tools.ConsumerOffsetChecker to view the consumer
> > > offset
> > > > > > > status, and can get the correct output, along with a
> > > > > > > warning: ConsumerOffsetChecker is deprecated and will be
> dropped
> > in
> > > > > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > > > > >
> > > > > > > But when I altered to bin/kafka-run-class.sh
> > > > > > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181
> > > --list,
> > > > > > there
> > > > > > > was nothing to output.
> > > > > > >
> > > > > > > Did I missed some parameters to run ConsumerGroupCommand? The
> > > version
> > > > > is
> > > > > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by Amoxicillin <sh...@gmail.com>.
I wrote the consumer by myself in golang using Shopify's sarama lib updated
of master branch.
On Fri, Feb 19, 2016 at 4:03 PM, tao xiao <xi...@gmail.com> wrote:
> which version of consumer do you use?
>
>
> On Fri, 19 Feb 2016 at 15:26 Amoxicillin <sh...@gmail.com> wrote:
>
> > I tried as you suggested, but still no output of any group info.
> >
> > On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > That is what I mean alive. If you use new consumer connecting to
> broker
> > > you should use --new-consumer option to list all consumer groups
> > >
> > > kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list
> --new-consumer
> > > --bootstrap-server localhost:9092
> > >
> > > On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com> wrote:
> > >
> > > > How to confirm the consumer groups are alive? I have one consumer in
> > the
> > > > group running at the same time, and could get messages correctly.
> > > >
> > > > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > when using ConsumerGroupCommand you need to make sure your consumer
> > > > groups
> > > > > are alive. It only queries offsets for consumer groups that are
> > > currently
> > > > > connecting to brokers
> > > > >
> > > > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I use kafka.tools.ConsumerOffsetChecker to view the consumer
> > offset
> > > > > > status, and can get the correct output, along with a
> > > > > > warning: ConsumerOffsetChecker is deprecated and will be dropped
> in
> > > > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > > > >
> > > > > > But when I altered to bin/kafka-run-class.sh
> > > > > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181
> > --list,
> > > > > there
> > > > > > was nothing to output.
> > > > > >
> > > > > > Did I missed some parameters to run ConsumerGroupCommand? The
> > version
> > > > is
> > > > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by tao xiao <xi...@gmail.com>.
which version of consumer do you use?
On Fri, 19 Feb 2016 at 15:26 Amoxicillin <sh...@gmail.com> wrote:
> I tried as you suggested, but still no output of any group info.
>
> On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com> wrote:
>
> > That is what I mean alive. If you use new consumer connecting to broker
> > you should use --new-consumer option to list all consumer groups
> >
> > kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --new-consumer
> > --bootstrap-server localhost:9092
> >
> > On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com> wrote:
> >
> > > How to confirm the consumer groups are alive? I have one consumer in
> the
> > > group running at the same time, and could get messages correctly.
> > >
> > > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > when using ConsumerGroupCommand you need to make sure your consumer
> > > groups
> > > > are alive. It only queries offsets for consumer groups that are
> > currently
> > > > connecting to brokers
> > > >
> > > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I use kafka.tools.ConsumerOffsetChecker to view the consumer
> offset
> > > > > status, and can get the correct output, along with a
> > > > > warning: ConsumerOffsetChecker is deprecated and will be dropped in
> > > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > > >
> > > > > But when I altered to bin/kafka-run-class.sh
> > > > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181
> --list,
> > > > there
> > > > > was nothing to output.
> > > > >
> > > > > Did I missed some parameters to run ConsumerGroupCommand? The
> version
> > > is
> > > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > > >
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by Amoxicillin <sh...@gmail.com>.
I tried as you suggested, but still no output of any group info.
On Fri, Feb 19, 2016 at 2:45 PM, tao xiao <xi...@gmail.com> wrote:
> That is what I mean alive. If you use new consumer connecting to broker
> you should use --new-consumer option to list all consumer groups
>
> kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --new-consumer
> --bootstrap-server localhost:9092
>
> On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com> wrote:
>
> > How to confirm the consumer groups are alive? I have one consumer in the
> > group running at the same time, and could get messages correctly.
> >
> > On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > when using ConsumerGroupCommand you need to make sure your consumer
> > groups
> > > are alive. It only queries offsets for consumer groups that are
> currently
> > > connecting to brokers
> > >
> > > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I use kafka.tools.ConsumerOffsetChecker to view the consumer offset
> > > > status, and can get the correct output, along with a
> > > > warning: ConsumerOffsetChecker is deprecated and will be dropped in
> > > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > > >
> > > > But when I altered to bin/kafka-run-class.sh
> > > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181 --list,
> > > there
> > > > was nothing to output.
> > > >
> > > > Did I missed some parameters to run ConsumerGroupCommand? The version
> > is
> > > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > > >
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by tao xiao <xi...@gmail.com>.
That is what I mean alive. If you use new consumer connecting to broker
you should use --new-consumer option to list all consumer groups
kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --new-consumer
--bootstrap-server localhost:9092
On Fri, 19 Feb 2016 at 14:18 Amoxicillin <sh...@gmail.com> wrote:
> How to confirm the consumer groups are alive? I have one consumer in the
> group running at the same time, and could get messages correctly.
>
> On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com> wrote:
>
> > when using ConsumerGroupCommand you need to make sure your consumer
> groups
> > are alive. It only queries offsets for consumer groups that are currently
> > connecting to brokers
> >
> > On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I use kafka.tools.ConsumerOffsetChecker to view the consumer offset
> > > status, and can get the correct output, along with a
> > > warning: ConsumerOffsetChecker is deprecated and will be dropped in
> > > releases following 0.9.0. Use ConsumerGroupCommand instead.
> > >
> > > But when I altered to bin/kafka-run-class.sh
> > > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181 --list,
> > there
> > > was nothing to output.
> > >
> > > Did I missed some parameters to run ConsumerGroupCommand? The version
> is
> > > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> > >
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by Amoxicillin <sh...@gmail.com>.
How to confirm the consumer groups are alive? I have one consumer in the
group running at the same time, and could get messages correctly.
On Fri, Feb 19, 2016 at 1:40 PM, tao xiao <xi...@gmail.com> wrote:
> when using ConsumerGroupCommand you need to make sure your consumer groups
> are alive. It only queries offsets for consumer groups that are currently
> connecting to brokers
>
> On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com> wrote:
>
> > Hi,
> >
> > I use kafka.tools.ConsumerOffsetChecker to view the consumer offset
> > status, and can get the correct output, along with a
> > warning: ConsumerOffsetChecker is deprecated and will be dropped in
> > releases following 0.9.0. Use ConsumerGroupCommand instead.
> >
> > But when I altered to bin/kafka-run-class.sh
> > kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181 --list,
> there
> > was nothing to output.
> >
> > Did I missed some parameters to run ConsumerGroupCommand? The version is
> > kafka _2.11-0.9.0.0, and any suggestion is appreciated.
> >
>
Re: ConsumerGroupCommand and ConsumerOffsetChecker
Posted by tao xiao <xi...@gmail.com>.
when using ConsumerGroupCommand you need to make sure your consumer groups
are alive. It only queries offsets for consumer groups that are currently
connecting to brokers
On Fri, 19 Feb 2016 at 13:35 Amoxicillin <sh...@gmail.com> wrote:
> Hi,
>
> I use kafka.tools.ConsumerOffsetChecker to view the consumer offset
> status, and can get the correct output, along with a
> warning: ConsumerOffsetChecker is deprecated and will be dropped in
> releases following 0.9.0. Use ConsumerGroupCommand instead.
>
> But when I altered to bin/kafka-run-class.sh
> kafka.admin.ConsumerGroupCommand --zookeeper 127.0.0.1:2181 --list, there
> was nothing to output.
>
> Did I missed some parameters to run ConsumerGroupCommand? The version is
> kafka _2.11-0.9.0.0, and any suggestion is appreciated.
>