You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zhuo Chen <cc...@gmail.com> on 2016/09/26 02:31:56 UTC
Where kafka stores consumer offsets in version 0.9.0.0?
hi, I use kafka 0.9.0.0 in CDH 5.6, and my application consume kafka
message by following code.
---------------------------------------------------------------------------------------------------------------------
String topicName = "tpCz01";
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.59.121:9092,
192.168.59.122:9092,192.168.59.123:9092");
props.put("group.id", "grpCz");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer
<String, String>(props);
//Kafka Consumer subscribes list of topics here.
consumer.subscribe(Arrays.asList(topicName));
//......
---------------------------------------------------------------------------------------------------------------------
when application run once, I can get consumer offset by
*kafka-run-class kafka.tools.ConsumerOffsetChecke, *but can not find the
offset in zookeeper path /consumer/xxxxx.
I wonder where kafka stores consumer offset in version 0.9.0.0, Is there
anything wrong I did.
any help would be appreciated. thank you~
Re: Where kafka stores consumer offsets in version 0.9.0.0?
Posted by Zhuo Chen <cc...@gmail.com>.
Kaufman, Thanks for your clear and helpful explanation.
The article you provided by link is definitely useful about new consumer
client, I have fully understood.
On Mon, Sep 26, 2016 at 10:36 PM, Kaufman Ng <ka...@confluent.io> wrote:
> Hi Zhuo,
>
> Since your code uses KafkaConsumer class, it's the "new consumer" in Kafka
> which uses a special kafka topic to keep track of offsets (rather than
> zookeeper). By default that topic is named "__consumer_offsets". You can
> use the "kafka-consumer-groups" command to check where the offset is for a
> given consumer group.
>
> You can find out more about the new consumer here:
> http://www.confluent.io/blog/tutorial-getting-started-with-
> the-new-apache-kafka-0.9-consumer-client/
>
> Hope this helps.
>
> On Sun, Sep 25, 2016 at 10:31 PM, Zhuo Chen <cc...@gmail.com> wrote:
>
> > hi, I use kafka 0.9.0.0 in CDH 5.6, and my application consume kafka
> > message by following code.
> >
> > ------------------------------------------------------------
> > ---------------------------------------------------------
> > String topicName = "tpCz01";
> > Properties props = new Properties();
> >
> > props.put("bootstrap.servers", "192.168.59.121:9092,
> > 192.168.59.122:9092,192.168.59.123:9092");
> > props.put("group.id", "grpCz");
> > props.put("enable.auto.commit", "true");
> > props.put("auto.commit.interval.ms", "1000");
> > props.put("session.timeout.ms", "30000");
> > props.put("key.deserializer",
> > "org.apache.kafka.common.serialization.StringDeserializer");
> > props.put("value.deserializer",
> > "org.apache.kafka.common.serialization.StringDeserializer");
> >
> > KafkaConsumer<String, String> consumer = new KafkaConsumer
> > <String, String>(props);
> >
> > //Kafka Consumer subscribes list of topics here.
> > consumer.subscribe(Arrays.asList(topicName));
> > //......
> > ------------------------------------------------------------
> > ---------------------------------------------------------
> >
> > when application run once, I can get consumer offset by
> > *kafka-run-class kafka.tools.ConsumerOffsetChecke, *but can not find the
> > offset in zookeeper path /consumer/xxxxx.
> >
> > I wonder where kafka stores consumer offset in version 0.9.0.0, Is there
> > anything wrong I did.
> > any help would be appreciated. thank you~
> >
>
>
>
> --
> Kaufman Ng
> Solutions Architect | Confluent
> +1 646 961 8063 | @kaufmanng
> www.confluent.io
>
Re: Where kafka stores consumer offsets in version 0.9.0.0?
Posted by Kaufman Ng <ka...@confluent.io>.
Hi Zhuo,
Since your code uses KafkaConsumer class, it's the "new consumer" in Kafka
which uses a special kafka topic to keep track of offsets (rather than
zookeeper). By default that topic is named "__consumer_offsets". You can
use the "kafka-consumer-groups" command to check where the offset is for a
given consumer group.
You can find out more about the new consumer here:
http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client/
Hope this helps.
On Sun, Sep 25, 2016 at 10:31 PM, Zhuo Chen <cc...@gmail.com> wrote:
> hi, I use kafka 0.9.0.0 in CDH 5.6, and my application consume kafka
> message by following code.
>
> ------------------------------------------------------------
> ---------------------------------------------------------
> String topicName = "tpCz01";
> Properties props = new Properties();
>
> props.put("bootstrap.servers", "192.168.59.121:9092,
> 192.168.59.122:9092,192.168.59.123:9092");
> props.put("group.id", "grpCz");
> props.put("enable.auto.commit", "true");
> props.put("auto.commit.interval.ms", "1000");
> props.put("session.timeout.ms", "30000");
> props.put("key.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
> props.put("value.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
>
> KafkaConsumer<String, String> consumer = new KafkaConsumer
> <String, String>(props);
>
> //Kafka Consumer subscribes list of topics here.
> consumer.subscribe(Arrays.asList(topicName));
> //......
> ------------------------------------------------------------
> ---------------------------------------------------------
>
> when application run once, I can get consumer offset by
> *kafka-run-class kafka.tools.ConsumerOffsetChecke, *but can not find the
> offset in zookeeper path /consumer/xxxxx.
>
> I wonder where kafka stores consumer offset in version 0.9.0.0, Is there
> anything wrong I did.
> any help would be appreciated. thank you~
>
--
Kaufman Ng
Solutions Architect | Confluent
+1 646 961 8063 | @kaufmanng
www.confluent.io