zookeeper usage in kafka offset commit / fetch requests

[svante karlsson <sa...@csi.se>]

I'm using kafka 0.8.2.0

I'm working on a C++ client library and I'm adding consumer offset
management to the client. (https://github.com/bitbouncer/csi-kafka)

I know that the creation of zookeeper "paths" is not handled by kafkabroker
so I've manually created

/consumers/consumer_offset_sample/offsets

in zookeeper using a command line utility.

After this I'm able to get consumer metadata from kafka.

If I commit a consumer offset to an existing topic/partition
("perf-8-new/0") I see the following paths in zookeeper

/consumers/consumer_offset_sample
offsets
offsets/perf-8-new
offsets/perf-8-new/0
owners

I'm surprised as to why the committed values shows up in zookeeper since I
have no bindings to a zookeeper and the "offset.storage" property is a
consumer config. My initial understanding was that they were only written
to __consumer_offsets topic.

Finally, I change the previously committed value manually in zookeeper
/consumers/consumer_offset_sample/offsets/perf-8-new/0  -> 42

then that's what I get back in my get_consumer_offset() as well - so it
seems the zookeeper is involved in the offsets commits/fetches from brokers
point of view.

Is this the intended way or am I doing something wrong?

best regards
svante

Re: zookeeper usage in kafka offset commit / fetch requests

[svante karlsson <sa...@csi.se>]

Just to close this,

I found out that the broker handles the request different depending on
whether you specify api version 0 or 1.

If using V1 the broker commits the offsets to internal topic.

/svante