Directly access consumer offsets from the special offset topic (__consumer_offsets)

[Ye Hong <Ye...@audiencescience.com>]

We are in the middle of moving to 0.8.2 and migrating consumer offsets from Zookeeper to Kafka is a major reason for the move.

After upgrading, we see a new topic being created:
Topic:__consumer_offsets	PartitionCount:50	ReplicationFactor:3	Configs:segment.bytes=xxx,cleanup.policy=compact

After passing a couple of messages through kafka mirror, I noticed that some of the partitions are changed:
__consumer_offsets-18:
total 0
-rw-r--r--. 1 kafka kafka xxx Nov  9 21:46 00000000000000000000.index
-rw-r--r--. 1 kafka kafka        0 Nov  9 21:46 00000000000000000000.log

__consumer_offsets-19:
total 1624
-rw-r--r--. 1 kafka kafka xxx Nov 10 01:14 00000000000000000000.index
-rw-r--r--. 1 kafka kafka  1652880 Nov 10 01:14 00000000000000000000.log

However, /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper zkhost --topic __consumer_offts --from-beginning yields no result at all.

Is it true that the offsets in special topic cannot be read directly like when they were in Zookeeper? We do have something that depends on direct access to consumer offsets. In addition, is it possible to reset the offsets in case of a mirror rewind. With Zookeeper, it is pretty easy to rewind the consumer offsets.

I am still able to check the mirror lags via kafka.tools.ConsumerOffsetChecker.

Thanks,

Ye