You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dhyan Muralidharan <d....@yottaa.com> on 2015/12/02 19:57:17 UTC
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
I have this same problem . Can someone help ?
--Dhyan
On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid> wrote:
> Hello,
>
> I'm trying to find out which offsets my current High-Level consumers are
> working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in the
> server.properties of Kafka - which, I think, means that offsets are stored
> in Kafka. (I also verified that no offsets are stored in Zookeeper by
> checking this path in the Zk shell:
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> )
>
> I tried to listen to the **__consumer_offsets** topic to see which
> consumer saves what value of offsets, but it did not work...
>
> I tried the following:
>
> created a config file for console consumer as following:
>
>
> => more kafka_offset_consumer.config
>
> exclude.internal.topics=false`
>
> and tried to versions of the console consumer scripts:
>
> #1:
> bin/kafka-console-consumer.sh --consumer.config
> kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> localhost:2181
>
> #2
> ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> --partition 0 --broker-list localhost:9092 --formatter
> "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
> kafka_offset_consumer.config
>
>
> Neither worked - it just sits there but does not print anything, even
> though the consumers are actively consuming/saving offsets.
>
> Am I missing some other configuration/properties ?
>
> thanks!
>
> Marina
>
> I have also posted this question on StackOverflow:
>
>
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
>
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Marina <pp...@yahoo.com.INVALID>.
Thank you, Lance - this is very useful info!I did figure out what was wrong in my case - the offsets were not stored in KAfka legitimately, they were stored in Zookeeper, I was using a wrong command to inspect ZK content - doing 'ls <path>' instead of 'get <path>'.
Once I used 'get' - I could see correct offsets for all my consumers in ZK. We will switch to using Kafka storage very soon!
Thanks!Marina
From: Lance Laursen <ll...@rubiconproject.com>
To: users@kafka.apache.org
Cc: Marina <pp...@yahoo.com>
Sent: Thursday, December 3, 2015 4:35 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Hi Marina,
You can hop onto your brokers and dump your __consumer_offsets logs
manually in order to see if anything is in them. Hop on each of your
brokers and run the following command:
for f in $(find /path/to/kafka-logs/__consumer_offsets-* -name "*\.log");
do /usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments
--print-data-log --files $f ; done
Remember to replace /path/to/kafka-logs with your configured log.dirs
location, and /usr/local/kafka/ with wherever you have installed kafka.
If that command doesn't generate any output (Starting offset: 0 followed by
nothing) on any of your brokers, then your consumers are for some reason
not committing offsets to Kafka.
Keep in mind that commits to __consumer_offsets by the consumer API use
groupID, Topic, and PartitionID to key off of in order to determine which
partition of __consumer_offsets to save the commit message to. This means
that if you're not running many consumers/topics, you could have partitions
with nothing in them and others with lots. Keep this in mind when using
-simple-consumer scripts as the simple consumer only pulls from one
partition at a time.
On Thu, Dec 3, 2015 at 11:10 AM, Guozhang Wang <wa...@gmail.com> wrote:
> If you can validate these partitions have data (i.e. there are some offsets
> committed to Kafka), then you may have to turn of debug level logging in
> configs/tools-log4j.properties which will allow console consumer to print
> debug level logs and see if there is anything suspicious.
>
> Guozhang
>
> On Thu, Dec 3, 2015 at 10:55 AM, Marina <pp...@yahoo.com.invalid> wrote:
>
> > Hi, Guozhang,
> > Yes, I can see this topic and partitions in ZK:
> >
> > ls /brokers/topics/__consumer_offsets
> > [partitions]
> > ls /brokers/topics/__consumer_offsets/partitions
> > [44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2,
> > 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
> >
> >
> > example of one partition state:
> > ls /brokers/topics/__consumer_offsets/partitions/44
> > [state]
> > ls /brokers/topics/__consumer_offsets/partitions/44/state
> > []
> >
> >
> > anything else you think I could check?
> >
> > thanks!
> > Marina
> >
> >
> >
> > ----- Original Message -----
> > From: Guozhang Wang <wa...@gmail.com>
> > To: "users@kafka.apache.org" <us...@kafka.apache.org>; Marina <
> > ppine7@yahoo.com>
> > Sent: Thursday, December 3, 2015 12:05 PM
> > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> >
> > Marina,
> >
> > To check if the topic does exist in Kafka (i.e. offsets are stored in
> Kafka
> > instead of in ZK) you can check this path in ZK:
> >
> > /brokers/topics/__consumer_offsets
> >
> > By default this topic should have 50 partitions.
> >
> > Guozhang
> >
> >
> >
> > On Thu, Dec 3, 2015 at 6:22 AM, Marina <pp...@yahoo.com.invalid> wrote:
> >
> > > Hi, Jason,
> > >
> > > I tried the same command both with specifying a formatter and without -
> > > same result:
> > >
> > > => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> > > /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> > > localhost:2181 --from-beginning
> > >
> > > ^CConsumed 0 messages
> > >
> > >
> > > When you are saying to make sure Consumers are using Kafka instead of
> ZK
> > > storage for offset... - what is the best way to do that? I thought
> > checking
> > > the content via ZK shell - at this path:
> > >
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
> > >
> > > would be a definitive confirmation (there are no values stored there).
> Is
> > > it true?
> > >
> > > I feel it is some small stupid mistake I'm making, since it seems to
> work
> > > fine for others - drives me crazy :)
> > >
> > > thanks!!
> > > Marina
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: users@kafka.apache.org
> > > Sent: Wednesday, December 2, 2015 2:33 PM
> > > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> > >
> > > Hey Marina,
> > >
> > > My mistake, I see you're using 0.8.2.1. Are you also providing the
> > > formatter argument when using console-consumer.sh? Perhaps something
> like
> > > this:
> > >
> > > bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> > > localhost:2181 --topic __consumer_offsets --from-beginning
> > >
> > > You may also want to confirm that your consumers are using Kafka
> instead
> > of
> > > Zookeeper for offset storage. If you still don't see anything, we can
> > > always look into the partition data directly...
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Looks like you need to use a different MessageFormatter class, since
> it
> > > > was renamed in 0.9. Instead use something like
> > > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > > > d.muralidharan@yottaa.com> wrote:
> > > >
> > > >> I have this same problem . Can someone help ?
> > > >>
> > > >> --Dhyan
> > > >>
> > > >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid>
> > > wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I'm trying to find out which offsets my current High-Level
> consumers
> > > are
> > > >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set
> > in
> > > >> the
> > > >> > server.properties of Kafka - which, I think, means that offsets
> are
> > > >> stored
> > > >> > in Kafka. (I also verified that no offsets are stored in Zookeeper
> > by
> > > >> > checking this path in the Zk shell:
> > > >> >
> > > >>
> > >
> >
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> > > >> > )
> > > >> >
> > > >> > I tried to listen to the **__consumer_offsets** topic to see which
> > > >> > consumer saves what value of offsets, but it did not work...
> > > >> >
> > > >> > I tried the following:
> > > >> >
> > > >> > created a config file for console consumer as following:
> > > >> >
> > > >> >
> > > >> > => more kafka_offset_consumer.config
> > > >> >
> > > >> > exclude.internal.topics=false`
> > > >> >
> > > >> > and tried to versions of the console consumer scripts:
> > > >> >
> > > >> > #1:
> > > >> > bin/kafka-console-consumer.sh --consumer.config
> > > >> > kafka_offset_consumer.config --topic __consumer_offsets
> --zookeeper
> > > >> > localhost:2181
> > > >> >
> > > >> > #2
> > > >> > ./bin/kafka-simple-consumer-shell.sh --topic
> __consumer_offsets
> > > >> > --partition 0 --broker-list localhost:9092 --formatter
> > > >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> > > --consumer.config
> > > >> > kafka_offset_consumer.config
> > > >> >
> > > >> >
> > > >> > Neither worked - it just sits there but does not print anything,
> > even
> > > >> > though the consumers are actively consuming/saving offsets.
> > > >> >
> > > >> > Am I missing some other configuration/properties ?
> > > >> >
> > > >> > thanks!
> > > >> >
> > > >> > Marina
> > > >> >
> > > >> > I have also posted this question on StackOverflow:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Lance Laursen <ll...@rubiconproject.com>.
Hi Marina,
You can hop onto your brokers and dump your __consumer_offsets logs
manually in order to see if anything is in them. Hop on each of your
brokers and run the following command:
for f in $(find /path/to/kafka-logs/__consumer_offsets-* -name "*\.log");
do /usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments
--print-data-log --files $f ; done
Remember to replace /path/to/kafka-logs with your configured log.dirs
location, and /usr/local/kafka/ with wherever you have installed kafka.
If that command doesn't generate any output (Starting offset: 0 followed by
nothing) on any of your brokers, then your consumers are for some reason
not committing offsets to Kafka.
Keep in mind that commits to __consumer_offsets by the consumer API use
groupID, Topic, and PartitionID to key off of in order to determine which
partition of __consumer_offsets to save the commit message to. This means
that if you're not running many consumers/topics, you could have partitions
with nothing in them and others with lots. Keep this in mind when using
-simple-consumer scripts as the simple consumer only pulls from one
partition at a time.
On Thu, Dec 3, 2015 at 11:10 AM, Guozhang Wang <wa...@gmail.com> wrote:
> If you can validate these partitions have data (i.e. there are some offsets
> committed to Kafka), then you may have to turn of debug level logging in
> configs/tools-log4j.properties which will allow console consumer to print
> debug level logs and see if there is anything suspicious.
>
> Guozhang
>
> On Thu, Dec 3, 2015 at 10:55 AM, Marina <pp...@yahoo.com.invalid> wrote:
>
> > Hi, Guozhang,
> > Yes, I can see this topic and partitions in ZK:
> >
> > ls /brokers/topics/__consumer_offsets
> > [partitions]
> > ls /brokers/topics/__consumer_offsets/partitions
> > [44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2,
> > 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
> >
> >
> > example of one partition state:
> > ls /brokers/topics/__consumer_offsets/partitions/44
> > [state]
> > ls /brokers/topics/__consumer_offsets/partitions/44/state
> > []
> >
> >
> > anything else you think I could check?
> >
> > thanks!
> > Marina
> >
> >
> >
> > ----- Original Message -----
> > From: Guozhang Wang <wa...@gmail.com>
> > To: "users@kafka.apache.org" <us...@kafka.apache.org>; Marina <
> > ppine7@yahoo.com>
> > Sent: Thursday, December 3, 2015 12:05 PM
> > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> >
> > Marina,
> >
> > To check if the topic does exist in Kafka (i.e. offsets are stored in
> Kafka
> > instead of in ZK) you can check this path in ZK:
> >
> > /brokers/topics/__consumer_offsets
> >
> > By default this topic should have 50 partitions.
> >
> > Guozhang
> >
> >
> >
> > On Thu, Dec 3, 2015 at 6:22 AM, Marina <pp...@yahoo.com.invalid> wrote:
> >
> > > Hi, Jason,
> > >
> > > I tried the same command both with specifying a formatter and without -
> > > same result:
> > >
> > > => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> > > /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> > > localhost:2181 --from-beginning
> > >
> > > ^CConsumed 0 messages
> > >
> > >
> > > When you are saying to make sure Consumers are using Kafka instead of
> ZK
> > > storage for offset... - what is the best way to do that? I thought
> > checking
> > > the content via ZK shell - at this path:
> > >
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
> > >
> > > would be a definitive confirmation (there are no values stored there).
> Is
> > > it true?
> > >
> > > I feel it is some small stupid mistake I'm making, since it seems to
> work
> > > fine for others - drives me crazy :)
> > >
> > > thanks!!
> > > Marina
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: users@kafka.apache.org
> > > Sent: Wednesday, December 2, 2015 2:33 PM
> > > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> > >
> > > Hey Marina,
> > >
> > > My mistake, I see you're using 0.8.2.1. Are you also providing the
> > > formatter argument when using console-consumer.sh? Perhaps something
> like
> > > this:
> > >
> > > bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> > > localhost:2181 --topic __consumer_offsets --from-beginning
> > >
> > > You may also want to confirm that your consumers are using Kafka
> instead
> > of
> > > Zookeeper for offset storage. If you still don't see anything, we can
> > > always look into the partition data directly...
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Looks like you need to use a different MessageFormatter class, since
> it
> > > > was renamed in 0.9. Instead use something like
> > > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > > > d.muralidharan@yottaa.com> wrote:
> > > >
> > > >> I have this same problem . Can someone help ?
> > > >>
> > > >> --Dhyan
> > > >>
> > > >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid>
> > > wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I'm trying to find out which offsets my current High-Level
> consumers
> > > are
> > > >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set
> > in
> > > >> the
> > > >> > server.properties of Kafka - which, I think, means that offsets
> are
> > > >> stored
> > > >> > in Kafka. (I also verified that no offsets are stored in Zookeeper
> > by
> > > >> > checking this path in the Zk shell:
> > > >> >
> > > >>
> > >
> >
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> > > >> > )
> > > >> >
> > > >> > I tried to listen to the **__consumer_offsets** topic to see which
> > > >> > consumer saves what value of offsets, but it did not work...
> > > >> >
> > > >> > I tried the following:
> > > >> >
> > > >> > created a config file for console consumer as following:
> > > >> >
> > > >> >
> > > >> > => more kafka_offset_consumer.config
> > > >> >
> > > >> > exclude.internal.topics=false`
> > > >> >
> > > >> > and tried to versions of the console consumer scripts:
> > > >> >
> > > >> > #1:
> > > >> > bin/kafka-console-consumer.sh --consumer.config
> > > >> > kafka_offset_consumer.config --topic __consumer_offsets
> --zookeeper
> > > >> > localhost:2181
> > > >> >
> > > >> > #2
> > > >> > ./bin/kafka-simple-consumer-shell.sh --topic
> __consumer_offsets
> > > >> > --partition 0 --broker-list localhost:9092 --formatter
> > > >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> > > --consumer.config
> > > >> > kafka_offset_consumer.config
> > > >> >
> > > >> >
> > > >> > Neither worked - it just sits there but does not print anything,
> > even
> > > >> > though the consumers are actively consuming/saving offsets.
> > > >> >
> > > >> > Am I missing some other configuration/properties ?
> > > >> >
> > > >> > thanks!
> > > >> >
> > > >> > Marina
> > > >> >
> > > >> > I have also posted this question on StackOverflow:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Guozhang Wang <wa...@gmail.com>.
If you can validate these partitions have data (i.e. there are some offsets
committed to Kafka), then you may have to turn of debug level logging in
configs/tools-log4j.properties which will allow console consumer to print
debug level logs and see if there is anything suspicious.
Guozhang
On Thu, Dec 3, 2015 at 10:55 AM, Marina <pp...@yahoo.com.invalid> wrote:
> Hi, Guozhang,
> Yes, I can see this topic and partitions in ZK:
>
> ls /brokers/topics/__consumer_offsets
> [partitions]
> ls /brokers/topics/__consumer_offsets/partitions
> [44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2,
> 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
>
>
> example of one partition state:
> ls /brokers/topics/__consumer_offsets/partitions/44
> [state]
> ls /brokers/topics/__consumer_offsets/partitions/44/state
> []
>
>
> anything else you think I could check?
>
> thanks!
> Marina
>
>
>
> ----- Original Message -----
> From: Guozhang Wang <wa...@gmail.com>
> To: "users@kafka.apache.org" <us...@kafka.apache.org>; Marina <
> ppine7@yahoo.com>
> Sent: Thursday, December 3, 2015 12:05 PM
> Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
>
> Marina,
>
> To check if the topic does exist in Kafka (i.e. offsets are stored in Kafka
> instead of in ZK) you can check this path in ZK:
>
> /brokers/topics/__consumer_offsets
>
> By default this topic should have 50 partitions.
>
> Guozhang
>
>
>
> On Thu, Dec 3, 2015 at 6:22 AM, Marina <pp...@yahoo.com.invalid> wrote:
>
> > Hi, Jason,
> >
> > I tried the same command both with specifying a formatter and without -
> > same result:
> >
> > => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> > kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> > /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> > localhost:2181 --from-beginning
> >
> > ^CConsumed 0 messages
> >
> >
> > When you are saying to make sure Consumers are using Kafka instead of ZK
> > storage for offset... - what is the best way to do that? I thought
> checking
> > the content via ZK shell - at this path:
> > /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
> >
> > would be a definitive confirmation (there are no values stored there). Is
> > it true?
> >
> > I feel it is some small stupid mistake I'm making, since it seems to work
> > fine for others - drives me crazy :)
> >
> > thanks!!
> > Marina
> >
> >
> >
> > ----- Original Message -----
> > From: Jason Gustafson <ja...@confluent.io>
> > To: users@kafka.apache.org
> > Sent: Wednesday, December 2, 2015 2:33 PM
> > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> >
> > Hey Marina,
> >
> > My mistake, I see you're using 0.8.2.1. Are you also providing the
> > formatter argument when using console-consumer.sh? Perhaps something like
> > this:
> >
> > bin/kafka-console-consumer.sh --formatter
> > kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> > localhost:2181 --topic __consumer_offsets --from-beginning
> >
> > You may also want to confirm that your consumers are using Kafka instead
> of
> > Zookeeper for offset storage. If you still don't see anything, we can
> > always look into the partition data directly...
> >
> > -Jason
> >
> >
> > On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Looks like you need to use a different MessageFormatter class, since it
> > > was renamed in 0.9. Instead use something like
> > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> > >
> > > -Jason
> > >
> > > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > > d.muralidharan@yottaa.com> wrote:
> > >
> > >> I have this same problem . Can someone help ?
> > >>
> > >> --Dhyan
> > >>
> > >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid>
> > wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I'm trying to find out which offsets my current High-Level consumers
> > are
> > >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set
> in
> > >> the
> > >> > server.properties of Kafka - which, I think, means that offsets are
> > >> stored
> > >> > in Kafka. (I also verified that no offsets are stored in Zookeeper
> by
> > >> > checking this path in the Zk shell:
> > >> >
> > >>
> >
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> > >> > )
> > >> >
> > >> > I tried to listen to the **__consumer_offsets** topic to see which
> > >> > consumer saves what value of offsets, but it did not work...
> > >> >
> > >> > I tried the following:
> > >> >
> > >> > created a config file for console consumer as following:
> > >> >
> > >> >
> > >> > => more kafka_offset_consumer.config
> > >> >
> > >> > exclude.internal.topics=false`
> > >> >
> > >> > and tried to versions of the console consumer scripts:
> > >> >
> > >> > #1:
> > >> > bin/kafka-console-consumer.sh --consumer.config
> > >> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> > >> > localhost:2181
> > >> >
> > >> > #2
> > >> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> > >> > --partition 0 --broker-list localhost:9092 --formatter
> > >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> > --consumer.config
> > >> > kafka_offset_consumer.config
> > >> >
> > >> >
> > >> > Neither worked - it just sits there but does not print anything,
> even
> > >> > though the consumers are actively consuming/saving offsets.
> > >> >
> > >> > Am I missing some other configuration/properties ?
> > >> >
> > >> > thanks!
> > >> >
> > >> > Marina
> > >> >
> > >> > I have also posted this question on StackOverflow:
> > >> >
> > >> >
> > >> >
> > >>
> >
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> > >> >
> > >>
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Marina <pp...@yahoo.com.INVALID>.
Hi, Guozhang,
Yes, I can see this topic and partitions in ZK:
ls /brokers/topics/__consumer_offsets
[partitions]
ls /brokers/topics/__consumer_offsets/partitions
[44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
example of one partition state:
ls /brokers/topics/__consumer_offsets/partitions/44
[state]
ls /brokers/topics/__consumer_offsets/partitions/44/state
[]
anything else you think I could check?
thanks!
Marina
----- Original Message -----
From: Guozhang Wang <wa...@gmail.com>
To: "users@kafka.apache.org" <us...@kafka.apache.org>; Marina <pp...@yahoo.com>
Sent: Thursday, December 3, 2015 12:05 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Marina,
To check if the topic does exist in Kafka (i.e. offsets are stored in Kafka
instead of in ZK) you can check this path in ZK:
/brokers/topics/__consumer_offsets
By default this topic should have 50 partitions.
Guozhang
On Thu, Dec 3, 2015 at 6:22 AM, Marina <pp...@yahoo.com.invalid> wrote:
> Hi, Jason,
>
> I tried the same command both with specifying a formatter and without -
> same result:
>
> => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> localhost:2181 --from-beginning
>
> ^CConsumed 0 messages
>
>
> When you are saying to make sure Consumers are using Kafka instead of ZK
> storage for offset... - what is the best way to do that? I thought checking
> the content via ZK shell - at this path:
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
>
> would be a definitive confirmation (there are no values stored there). Is
> it true?
>
> I feel it is some small stupid mistake I'm making, since it seems to work
> fine for others - drives me crazy :)
>
> thanks!!
> Marina
>
>
>
> ----- Original Message -----
> From: Jason Gustafson <ja...@confluent.io>
> To: users@kafka.apache.org
> Sent: Wednesday, December 2, 2015 2:33 PM
> Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
>
> Hey Marina,
>
> My mistake, I see you're using 0.8.2.1. Are you also providing the
> formatter argument when using console-consumer.sh? Perhaps something like
> this:
>
> bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> localhost:2181 --topic __consumer_offsets --from-beginning
>
> You may also want to confirm that your consumers are using Kafka instead of
> Zookeeper for offset storage. If you still don't see anything, we can
> always look into the partition data directly...
>
> -Jason
>
>
> On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Looks like you need to use a different MessageFormatter class, since it
> > was renamed in 0.9. Instead use something like
> > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> >
> > -Jason
> >
> > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > d.muralidharan@yottaa.com> wrote:
> >
> >> I have this same problem . Can someone help ?
> >>
> >> --Dhyan
> >>
> >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid>
> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm trying to find out which offsets my current High-Level consumers
> are
> >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
> >> the
> >> > server.properties of Kafka - which, I think, means that offsets are
> >> stored
> >> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
> >> > checking this path in the Zk shell:
> >> >
> >>
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> >> > )
> >> >
> >> > I tried to listen to the **__consumer_offsets** topic to see which
> >> > consumer saves what value of offsets, but it did not work...
> >> >
> >> > I tried the following:
> >> >
> >> > created a config file for console consumer as following:
> >> >
> >> >
> >> > => more kafka_offset_consumer.config
> >> >
> >> > exclude.internal.topics=false`
> >> >
> >> > and tried to versions of the console consumer scripts:
> >> >
> >> > #1:
> >> > bin/kafka-console-consumer.sh --consumer.config
> >> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> >> > localhost:2181
> >> >
> >> > #2
> >> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> >> > --partition 0 --broker-list localhost:9092 --formatter
> >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> --consumer.config
> >> > kafka_offset_consumer.config
> >> >
> >> >
> >> > Neither worked - it just sits there but does not print anything, even
> >> > though the consumers are actively consuming/saving offsets.
> >> >
> >> > Am I missing some other configuration/properties ?
> >> >
> >> > thanks!
> >> >
> >> > Marina
> >> >
> >> > I have also posted this question on StackOverflow:
> >> >
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> >> >
> >>
> >
> >
>
--
-- Guozhang
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Guozhang Wang <wa...@gmail.com>.
Marina,
To check if the topic does exist in Kafka (i.e. offsets are stored in Kafka
instead of in ZK) you can check this path in ZK:
/brokers/topics/__consumer_offsets
By default this topic should have 50 partitions.
Guozhang
On Thu, Dec 3, 2015 at 6:22 AM, Marina <pp...@yahoo.com.invalid> wrote:
> Hi, Jason,
>
> I tried the same command both with specifying a formatter and without -
> same result:
>
> => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> localhost:2181 --from-beginning
>
> ^CConsumed 0 messages
>
>
> When you are saying to make sure Consumers are using Kafka instead of ZK
> storage for offset... - what is the best way to do that? I thought checking
> the content via ZK shell - at this path:
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
>
> would be a definitive confirmation (there are no values stored there). Is
> it true?
>
> I feel it is some small stupid mistake I'm making, since it seems to work
> fine for others - drives me crazy :)
>
> thanks!!
> Marina
>
>
>
> ----- Original Message -----
> From: Jason Gustafson <ja...@confluent.io>
> To: users@kafka.apache.org
> Sent: Wednesday, December 2, 2015 2:33 PM
> Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
>
> Hey Marina,
>
> My mistake, I see you're using 0.8.2.1. Are you also providing the
> formatter argument when using console-consumer.sh? Perhaps something like
> this:
>
> bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> localhost:2181 --topic __consumer_offsets --from-beginning
>
> You may also want to confirm that your consumers are using Kafka instead of
> Zookeeper for offset storage. If you still don't see anything, we can
> always look into the partition data directly...
>
> -Jason
>
>
> On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Looks like you need to use a different MessageFormatter class, since it
> > was renamed in 0.9. Instead use something like
> > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> >
> > -Jason
> >
> > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > d.muralidharan@yottaa.com> wrote:
> >
> >> I have this same problem . Can someone help ?
> >>
> >> --Dhyan
> >>
> >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid>
> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm trying to find out which offsets my current High-Level consumers
> are
> >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
> >> the
> >> > server.properties of Kafka - which, I think, means that offsets are
> >> stored
> >> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
> >> > checking this path in the Zk shell:
> >> >
> >>
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> >> > )
> >> >
> >> > I tried to listen to the **__consumer_offsets** topic to see which
> >> > consumer saves what value of offsets, but it did not work...
> >> >
> >> > I tried the following:
> >> >
> >> > created a config file for console consumer as following:
> >> >
> >> >
> >> > => more kafka_offset_consumer.config
> >> >
> >> > exclude.internal.topics=false`
> >> >
> >> > and tried to versions of the console consumer scripts:
> >> >
> >> > #1:
> >> > bin/kafka-console-consumer.sh --consumer.config
> >> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> >> > localhost:2181
> >> >
> >> > #2
> >> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> >> > --partition 0 --broker-list localhost:9092 --formatter
> >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> --consumer.config
> >> > kafka_offset_consumer.config
> >> >
> >> >
> >> > Neither worked - it just sits there but does not print anything, even
> >> > though the consumers are actively consuming/saving offsets.
> >> >
> >> > Am I missing some other configuration/properties ?
> >> >
> >> > thanks!
> >> >
> >> > Marina
> >> >
> >> > I have also posted this question on StackOverflow:
> >> >
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> >> >
> >>
> >
> >
>
--
-- Guozhang
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Marina <pp...@yahoo.com.INVALID>.
Hi, Jason,
I tried the same command both with specifying a formatter and without - same result:
=> /opt/kafka/bin/kafka-console-consumer.sh --formatter kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config /tmp/consumer.properties --topic __consumer_offsets --zookeeper localhost:2181 --from-beginning
^CConsumed 0 messages
When you are saying to make sure Consumers are using Kafka instead of ZK storage for offset... - what is the best way to do that? I thought checking the content via ZK shell - at this path:
/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
would be a definitive confirmation (there are no values stored there). Is it true?
I feel it is some small stupid mistake I'm making, since it seems to work fine for others - drives me crazy :)
thanks!!
Marina
----- Original Message -----
From: Jason Gustafson <ja...@confluent.io>
To: users@kafka.apache.org
Sent: Wednesday, December 2, 2015 2:33 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Hey Marina,
My mistake, I see you're using 0.8.2.1. Are you also providing the
formatter argument when using console-consumer.sh? Perhaps something like
this:
bin/kafka-console-consumer.sh --formatter
kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
localhost:2181 --topic __consumer_offsets --from-beginning
You may also want to confirm that your consumers are using Kafka instead of
Zookeeper for offset storage. If you still don't see anything, we can
always look into the partition data directly...
-Jason
On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Looks like you need to use a different MessageFormatter class, since it
> was renamed in 0.9. Instead use something like
> "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
>
> -Jason
>
> On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> d.muralidharan@yottaa.com> wrote:
>
>> I have this same problem . Can someone help ?
>>
>> --Dhyan
>>
>> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid> wrote:
>>
>> > Hello,
>> >
>> > I'm trying to find out which offsets my current High-Level consumers are
>> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
>> the
>> > server.properties of Kafka - which, I think, means that offsets are
>> stored
>> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
>> > checking this path in the Zk shell:
>> >
>> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
>> > )
>> >
>> > I tried to listen to the **__consumer_offsets** topic to see which
>> > consumer saves what value of offsets, but it did not work...
>> >
>> > I tried the following:
>> >
>> > created a config file for console consumer as following:
>> >
>> >
>> > => more kafka_offset_consumer.config
>> >
>> > exclude.internal.topics=false`
>> >
>> > and tried to versions of the console consumer scripts:
>> >
>> > #1:
>> > bin/kafka-console-consumer.sh --consumer.config
>> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
>> > localhost:2181
>> >
>> > #2
>> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
>> > --partition 0 --broker-list localhost:9092 --formatter
>> > "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
>> > kafka_offset_consumer.config
>> >
>> >
>> > Neither worked - it just sits there but does not print anything, even
>> > though the consumers are actively consuming/saving offsets.
>> >
>> > Am I missing some other configuration/properties ?
>> >
>> > thanks!
>> >
>> > Marina
>> >
>> > I have also posted this question on StackOverflow:
>> >
>> >
>> >
>> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
>> >
>>
>
>
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Marina,
My mistake, I see you're using 0.8.2.1. Are you also providing the
formatter argument when using console-consumer.sh? Perhaps something like
this:
bin/kafka-console-consumer.sh --formatter
kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
localhost:2181 --topic __consumer_offsets --from-beginning
You may also want to confirm that your consumers are using Kafka instead of
Zookeeper for offset storage. If you still don't see anything, we can
always look into the partition data directly...
-Jason
On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Looks like you need to use a different MessageFormatter class, since it
> was renamed in 0.9. Instead use something like
> "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
>
> -Jason
>
> On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> d.muralidharan@yottaa.com> wrote:
>
>> I have this same problem . Can someone help ?
>>
>> --Dhyan
>>
>> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid> wrote:
>>
>> > Hello,
>> >
>> > I'm trying to find out which offsets my current High-Level consumers are
>> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
>> the
>> > server.properties of Kafka - which, I think, means that offsets are
>> stored
>> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
>> > checking this path in the Zk shell:
>> >
>> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
>> > )
>> >
>> > I tried to listen to the **__consumer_offsets** topic to see which
>> > consumer saves what value of offsets, but it did not work...
>> >
>> > I tried the following:
>> >
>> > created a config file for console consumer as following:
>> >
>> >
>> > => more kafka_offset_consumer.config
>> >
>> > exclude.internal.topics=false`
>> >
>> > and tried to versions of the console consumer scripts:
>> >
>> > #1:
>> > bin/kafka-console-consumer.sh --consumer.config
>> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
>> > localhost:2181
>> >
>> > #2
>> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
>> > --partition 0 --broker-list localhost:9092 --formatter
>> > "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
>> > kafka_offset_consumer.config
>> >
>> >
>> > Neither worked - it just sits there but does not print anything, even
>> > though the consumers are actively consuming/saving offsets.
>> >
>> > Am I missing some other configuration/properties ?
>> >
>> > thanks!
>> >
>> > Marina
>> >
>> > I have also posted this question on StackOverflow:
>> >
>> >
>> >
>> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
>> >
>>
>
>
Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Posted by Jason Gustafson <ja...@confluent.io>.
Looks like you need to use a different MessageFormatter class, since it was
renamed in 0.9. Instead use something like
"kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
-Jason
On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
d.muralidharan@yottaa.com> wrote:
> I have this same problem . Can someone help ?
>
> --Dhyan
>
> On Wed, Nov 25, 2015 at 3:31 PM, Marina <pp...@yahoo.com.invalid> wrote:
>
> > Hello,
> >
> > I'm trying to find out which offsets my current High-Level consumers are
> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in the
> > server.properties of Kafka - which, I think, means that offsets are
> stored
> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
> > checking this path in the Zk shell:
> >
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> > )
> >
> > I tried to listen to the **__consumer_offsets** topic to see which
> > consumer saves what value of offsets, but it did not work...
> >
> > I tried the following:
> >
> > created a config file for console consumer as following:
> >
> >
> > => more kafka_offset_consumer.config
> >
> > exclude.internal.topics=false`
> >
> > and tried to versions of the console consumer scripts:
> >
> > #1:
> > bin/kafka-console-consumer.sh --consumer.config
> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> > localhost:2181
> >
> > #2
> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> > --partition 0 --broker-list localhost:9092 --formatter
> > "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
> > kafka_offset_consumer.config
> >
> >
> > Neither worked - it just sits there but does not print anything, even
> > though the consumers are actively consuming/saving offsets.
> >
> > Am I missing some other configuration/properties ?
> >
> > thanks!
> >
> > Marina
> >
> > I have also posted this question on StackOverflow:
> >
> >
> >
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> >
>