Storing offsets in ZooKeeper for a SimpleConsumer

[Buck Ryan <fr...@rjmetrics.com>]

Hi,

We are writing our own consumer implementation and need to store offsets
manually in ZooKeeper. Currently we are storing them in a top-level znode
with a structure similar to what is found in /consumers. For example, we
have

/[consumer-name]/offsets/partition/[partition-id]

In looking into some monitoring tools like Burrow and KafkaOffsetMonitor,
they seem to only be geared toward the high-level consumer. So my question
is - is the recommended practice for writing one's own consumer to still
store offsets within the /consumers znode? Ie, store my offsets in
/consumers/[consumer-name]/offsets/partition/[partition-id] despite not
actually using the high-level consumer?

My concerns for doing so are whether Kafka makes assumptions about data
within the /consumers znode such that it would cause issues to store data
in there.

-- 

*Buck Ryan*
Software Engineer

1339 Chestnut Street, Ste 1500
Philadelphia, PA 19107

Website <http://rjmetrics.com>   |   Email <fr...@rjmetrics.com>   |
Twitter <http://twitter.com/_buckryan>

Re: Storing offsets in ZooKeeper for a SimpleConsumer

["Helleren, Erik" <Er...@cmegroup.com>]

Hi Buck,
So, if you are on 0.8.2, you can use the SimpleConsumer¹s commitOffsets
function which allows you to commit offsets to either zookeeper or a kafka
topic.

I would avoid rolling your own solution that edits data within existing
zookeeper paths because it might make upgrading to future versions of
kafka more difficult.  With that said, I can¹t say what kafka expects to
be at those nodes so it might be safe so long as they don¹t exist already.
 

If you are using another language or doing this from the ground up, I
would look at how the code in commitOffsets does it.  If that were the
case, I would use a path of group/topic/partition/offset And not use the
name of an individual consumer.  This is to preserve the semantic of
consumer groups which is very useful for clustering consumers.
-Erik

On 8/25/15, 8:58 AM, "Buck Ryan" <fr...@rjmetrics.com> wrote:

>Hi,
>
>We are writing our own consumer implementation and need to store offsets
>manually in ZooKeeper. Currently we are storing them in a top-level znode
>with a structure similar to what is found in /consumers. For example, we
>have
>
>/[consumer-name]/offsets/partition/[partition-id]
>
>In looking into some monitoring tools like Burrow and KafkaOffsetMonitor,
>they seem to only be geared toward the high-level consumer. So my question
>is - is the recommended practice for writing one's own consumer to still
>store offsets within the /consumers znode? Ie, store my offsets in
>/consumers/[consumer-name]/offsets/partition/[partition-id] despite not
>actually using the high-level consumer?
>
>My concerns for doing so are whether Kafka makes assumptions about data
>within the /consumers znode such that it would cause issues to store data
>in there.
>
>-- 
>
>*Buck Ryan*
>Software Engineer
>
>1339 Chestnut Street, Ste 1500
>Philadelphia, PA 19107
>
>Website <http://rjmetrics.com>   |   Email <fr...@rjmetrics.com>   |
>Twitter <http://twitter.com/_buckryan>