You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by karan alang <ka...@gmail.com> on 2017/06/15 18:55:00 UTC
kafka-consumer-groups.sh vs ConsumerOffsetChecker
Hi All -
I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to
use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for
offsets.
I'm seeing different behavior.
Here is what i did ->
a) When i use ConsumerOffsetChecker
$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--zookeeper localhost:2181 --group myGroup --topic newBroker1
[2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is deprecated
and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand
instead. (kafka.tools.ConsumerOffsetChecker$)
Group Topic Pid Offset logSize
Lag Owner
myGroup newBroker1 0 224216 507732
283516 none
myGroup newBroker1 1 224888 508978
284090 none
myGroup newBroker1 2 141104 424038
282934 none
myGroup newBroker1 3 11829 295110
283281 none
myGroup newBroker1 4 11580 294510
282930 none
b) When i run the kafka-consumer-groups.sh (since the WARNING message above
says - ConsumerOffsetChecker is deprecated & we need to use
ConsumerGroupCommand instead)
Option 1 :
$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server
localhost:9092,localhost:9093,localhost:9094,localhost:9095 --new-consumer
--group myGroup
Consumer group `myGroup` does not exist or is rebalancing.
The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why is
the script not giving me Offset details of the group - myGroup & Topic -
newBroker1
What needs to be done for this (using script
$KAFKA_HOME/bin/kafka-consumer-groups.sh) ?
Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Is it possible that the consumer group or the corresponding topic got
deleted?
If you don't have consumers running in the group but there are offsets
associated with (valid) topics the group, group offsets should not get
removed from ZK.
--Vahid
From: karan alang <ka...@gmail.com>
To: users@kafka.apache.org
Date: 06/15/2017 02:22 PM
Subject: Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Hi Vahid,
In ZK, i dont see any reference to myGroup
(is this because the consumer is no longer running or is there some other
reason ?)
*ls /consumers[replay-log-producer]*
Also, when i run the ConsumerOffsetChecker w/o the topic, i get error
shown
below ->
*$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--zookeeper localhost:2181 --group myGroup*
[2017-06-15 14:12:35,121] WARN WARNING: ConsumerOffsetChecker is
deprecated
and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand
instead. (kafka.tools.ConsumerOffsetChecker$)
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/myGroup/owners.
Btw, even with the --topic, i'm getting this error now.
(which i was not getting earlier)
So, How do i interpret this ?
is this a caching issue ?
On Thu, Jun 15, 2017 at 1:01 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Karan,
>
> The message "No topic available for consumer group provided" appears
when
> there is no topic under the consumer group in ZooKeeper (under
> /consumers/myGroup/owners/).
> Can you check whether the topic 'newBroker1' exists under this ZK path?
> Also, do you still get the rows below if you run the
ConsumerOffsetChecker
> without providing any topic?
>
> Thanks.
> --Vahid
>
>
>
>
> From: karan alang <ka...@gmail.com>
> To: users@kafka.apache.org
> Date: 06/15/2017 12:10 PM
> Subject: Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
>
>
>
> Pls note ->
> Even when i run the command as shown below (not as new-consumer), i
don't
> get the required result.
>
> *$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper
> localhost:2181 --group myGroup*
>
> No topic available for consumer group provided
>
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
>
> any ideas on what needs to be done?
>
> On Thu, Jun 15, 2017 at 11:55 AM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hi All -
> > I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) &
trying
> to
> > use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check
> for
> > offsets.
> >
> > I'm seeing different behavior.
> >
> > Here is what i did ->
> >
> > a) When i use ConsumerOffsetChecker
> >
> > $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > --zookeeper localhost:2181 --group myGroup --topic newBroker1
> >
> > [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is
> > deprecated and will be dropped in releases following 0.9.0. Use
> > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> >
> > Group Topic Pid Offset logSize
> > Lag Owner
> >
> > myGroup newBroker1 0 224216 507732
> > 283516 none
> >
> > myGroup newBroker1 1 224888 508978
> > 284090 none
> >
> > myGroup newBroker1 2 141104 424038
> > 282934 none
> >
> > myGroup newBroker1 3 11829 295110
> > 283281 none
> >
> > myGroup newBroker1 4 11580 294510
> > 282930 none
> >
> > b) When i run the kafka-consumer-groups.sh (since the WARNING message
> > above says - ConsumerOffsetChecker is deprecated & we need to use
> > ConsumerGroupCommand instead)
> >
> > Option 1 :
> >
> > $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server
> > localhost:9092,localhost:9093,localhost:9094,localhost:9095
> > --new-consumer --group myGroup
> >
> > Consumer group `myGroup` does not exist or is rebalancing.
> >
> >
> > The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh -
why
> > is the script not giving me Offset details of the group - myGroup &
> Topic -
> > newBroker1
> >
> > What needs to be done for this (using script
> $KAFKA_HOME/bin/kafka-consumer-groups.sh)
> > ?
> >
> >
> >
> >
>
>
>
>
>
Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Posted by karan alang <ka...@gmail.com>.
Hi Vahid,
In ZK, i dont see any reference to myGroup
(is this because the consumer is no longer running or is there some other
reason ?)
*ls /consumers[replay-log-producer]*
Also, when i run the ConsumerOffsetChecker w/o the topic, i get error shown
below ->
*$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--zookeeper localhost:2181 --group myGroup*
[2017-06-15 14:12:35,121] WARN WARNING: ConsumerOffsetChecker is deprecated
and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand
instead. (kafka.tools.ConsumerOffsetChecker$)
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/myGroup/owners.
Btw, even with the --topic, i'm getting this error now.
(which i was not getting earlier)
So, How do i interpret this ?
is this a caching issue ?
On Thu, Jun 15, 2017 at 1:01 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Karan,
>
> The message "No topic available for consumer group provided" appears when
> there is no topic under the consumer group in ZooKeeper (under
> /consumers/myGroup/owners/).
> Can you check whether the topic 'newBroker1' exists under this ZK path?
> Also, do you still get the rows below if you run the ConsumerOffsetChecker
> without providing any topic?
>
> Thanks.
> --Vahid
>
>
>
>
> From: karan alang <ka...@gmail.com>
> To: users@kafka.apache.org
> Date: 06/15/2017 12:10 PM
> Subject: Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
>
>
>
> Pls note ->
> Even when i run the command as shown below (not as new-consumer), i don't
> get the required result.
>
> *$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper
> localhost:2181 --group myGroup*
>
> No topic available for consumer group provided
>
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
>
> any ideas on what needs to be done?
>
> On Thu, Jun 15, 2017 at 11:55 AM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hi All -
> > I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying
> to
> > use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check
> for
> > offsets.
> >
> > I'm seeing different behavior.
> >
> > Here is what i did ->
> >
> > a) When i use ConsumerOffsetChecker
> >
> > $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > --zookeeper localhost:2181 --group myGroup --topic newBroker1
> >
> > [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is
> > deprecated and will be dropped in releases following 0.9.0. Use
> > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> >
> > Group Topic Pid Offset logSize
> > Lag Owner
> >
> > myGroup newBroker1 0 224216 507732
> > 283516 none
> >
> > myGroup newBroker1 1 224888 508978
> > 284090 none
> >
> > myGroup newBroker1 2 141104 424038
> > 282934 none
> >
> > myGroup newBroker1 3 11829 295110
> > 283281 none
> >
> > myGroup newBroker1 4 11580 294510
> > 282930 none
> >
> > b) When i run the kafka-consumer-groups.sh (since the WARNING message
> > above says - ConsumerOffsetChecker is deprecated & we need to use
> > ConsumerGroupCommand instead)
> >
> > Option 1 :
> >
> > $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server
> > localhost:9092,localhost:9093,localhost:9094,localhost:9095
> > --new-consumer --group myGroup
> >
> > Consumer group `myGroup` does not exist or is rebalancing.
> >
> >
> > The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why
> > is the script not giving me Offset details of the group - myGroup &
> Topic -
> > newBroker1
> >
> > What needs to be done for this (using script
> $KAFKA_HOME/bin/kafka-consumer-groups.sh)
> > ?
> >
> >
> >
> >
>
>
>
>
>
Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Karan,
The message "No topic available for consumer group provided" appears when
there is no topic under the consumer group in ZooKeeper (under
/consumers/myGroup/owners/).
Can you check whether the topic 'newBroker1' exists under this ZK path?
Also, do you still get the rows below if you run the ConsumerOffsetChecker
without providing any topic?
Thanks.
--Vahid
From: karan alang <ka...@gmail.com>
To: users@kafka.apache.org
Date: 06/15/2017 12:10 PM
Subject: Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Pls note ->
Even when i run the command as shown below (not as new-consumer), i don't
get the required result.
*$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper
localhost:2181 --group myGroup*
No topic available for consumer group provided
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
any ideas on what needs to be done?
On Thu, Jun 15, 2017 at 11:55 AM, karan alang <ka...@gmail.com>
wrote:
> Hi All -
> I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying
to
> use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check
for
> offsets.
>
> I'm seeing different behavior.
>
> Here is what i did ->
>
> a) When i use ConsumerOffsetChecker
>
> $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> --zookeeper localhost:2181 --group myGroup --topic newBroker1
>
> [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is
> deprecated and will be dropped in releases following 0.9.0. Use
> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
>
> Group Topic Pid Offset logSize
> Lag Owner
>
> myGroup newBroker1 0 224216 507732
> 283516 none
>
> myGroup newBroker1 1 224888 508978
> 284090 none
>
> myGroup newBroker1 2 141104 424038
> 282934 none
>
> myGroup newBroker1 3 11829 295110
> 283281 none
>
> myGroup newBroker1 4 11580 294510
> 282930 none
>
> b) When i run the kafka-consumer-groups.sh (since the WARNING message
> above says - ConsumerOffsetChecker is deprecated & we need to use
> ConsumerGroupCommand instead)
>
> Option 1 :
>
> $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server
> localhost:9092,localhost:9093,localhost:9094,localhost:9095
> --new-consumer --group myGroup
>
> Consumer group `myGroup` does not exist or is rebalancing.
>
>
> The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why
> is the script not giving me Offset details of the group - myGroup &
Topic -
> newBroker1
>
> What needs to be done for this (using script
$KAFKA_HOME/bin/kafka-consumer-groups.sh)
> ?
>
>
>
>
Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker
Posted by karan alang <ka...@gmail.com>.
Pls note ->
Even when i run the command as shown below (not as new-consumer), i don't
get the required result.
*$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper
localhost:2181 --group myGroup*
No topic available for consumer group provided
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
any ideas on what needs to be done?
On Thu, Jun 15, 2017 at 11:55 AM, karan alang <ka...@gmail.com> wrote:
> Hi All -
> I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to
> use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for
> offsets.
>
> I'm seeing different behavior.
>
> Here is what i did ->
>
> a) When i use ConsumerOffsetChecker
>
> $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> --zookeeper localhost:2181 --group myGroup --topic newBroker1
>
> [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is
> deprecated and will be dropped in releases following 0.9.0. Use
> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
>
> Group Topic Pid Offset logSize
> Lag Owner
>
> myGroup newBroker1 0 224216 507732
> 283516 none
>
> myGroup newBroker1 1 224888 508978
> 284090 none
>
> myGroup newBroker1 2 141104 424038
> 282934 none
>
> myGroup newBroker1 3 11829 295110
> 283281 none
>
> myGroup newBroker1 4 11580 294510
> 282930 none
>
> b) When i run the kafka-consumer-groups.sh (since the WARNING message
> above says - ConsumerOffsetChecker is deprecated & we need to use
> ConsumerGroupCommand instead)
>
> Option 1 :
>
> $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server
> localhost:9092,localhost:9093,localhost:9094,localhost:9095
> --new-consumer --group myGroup
>
> Consumer group `myGroup` does not exist or is rebalancing.
>
>
> The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why
> is the script not giving me Offset details of the group - myGroup & Topic -
> newBroker1
>
> What needs to be done for this (using script $KAFKA_HOME/bin/kafka-consumer-groups.sh)
> ?
>
>
>
>