kafka 0.10 offset storage on zookeeper

[Wei Li <we...@gmail.com>]

Hi All,

We recently upgraded kafka from 0.8 to 0.10 with security features enabled
(ssl/sasl/acls).  One thing noticed is the offset is not longer kept on
zookeeper any more in 0.10 release.  Though this is new feature for new
version, we like to still keep the offsets stored on zookeeper in short
terms without breaking the scripts/monitoring utilities that use old style
offset.  By following the documentation, I have enabled
dual.commit.enabled=true and offset.storage=zookeeper.  After rolling
restarted, the offsets are still not saved in zookeeper.  There is no
errors recorded in server or zookeeper logs.   Can someone suggest how I
can further debug this issue?   BTW, I am using logstash as kafka client to
read data from kafka topic.

Thanks,
Wei

Re: kafka 0.10 offset storage on zookeeper

[Guozhang Wang <wa...@gmail.com>]

I think Onur has once mentioned similar issues that currently we do not
have a fully automated smooth upgrade path from old consumer (offsets
stored in ZK) to new consumer (offsets stored in Kafka), maybe he can
provide more context here and how it can be resolved.

Currently the following procedure would be a not-perfect mechanism I can
think of:

1. Upgrade to 0.8.2 first, turn on dual.commit.enabled and offset.storage
to zookeeper, ran for a while until you can make sure that at least one set
of offsets has been written to brokers.

2. Stop all consumers, and the restart them with the newer version. Note
that you cannot do rolling bounce to keep the application live during this
upgrade since during the time there are both old and new consumers, since
they will be coordinated differently reading offsets from different places
(ZK v.s. brokers), they could potentially fetching the same partitions and
hence cause duplicate data.



Guozhang




On Sun, Nov 13, 2016 at 5:45 PM, Wei Li <we...@gmail.com> wrote:

> Hi All,
>
> We recently upgraded kafka from 0.8 to 0.10 with security features enabled
> (ssl/sasl/acls).  One thing noticed is the offset is not longer kept on
> zookeeper any more in 0.10 release.  Though this is new feature for new
> version, we like to still keep the offsets stored on zookeeper in short
> terms without breaking the scripts/monitoring utilities that use old style
> offset.  By following the documentation, I have enabled
> dual.commit.enabled=true and offset.storage=zookeeper.  After rolling
> restarted, the offsets are still not saved in zookeeper.  There is no
> errors recorded in server or zookeeper logs.   Can someone suggest how I
> can further debug this issue?   BTW, I am using logstash as kafka client to
> read data from kafka topic.
>
> Thanks,
> Wei
>



-- 
-- Guozhang