You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kris K <sq...@gmail.com> on 2015/06/11 10:03:09 UTC
offset storage as kafka with zookeeper 3.4.6
I am trying to migrate the offset storage to kafka (3 brokers of version
0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that
a new topic, __consumer_offsets got created.
But nothing is being written to this topic, while the consumer offsets
continue to reside on zookeeper.
I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
one that comes with kafka.
The current config consumer.properties now contains:
offsets.storage=kafka
dual.commit.enabled=false
exclude.internal.topics=false
Is it mandatory to use the zookeeper that comes with kafka for offset
storage to be migrated to kafka?
I tried both the approaches:
1. As listed on slide 34 of
http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
2. By deleting the zookeeper data directories and kafka log directories.
None of them worked.
Thanks
Kris
Re: offset storage as kafka with zookeeper 3.4.6
Posted by Kris K <sq...@gmail.com>.
>If you want to "move" offsets from zookeeper to Kafka then yes you
>need to have a phase where all consumers in your group set dual commit
>to true. If you are starting a fresh consumer group then you can
>turn off dual-commit.
I followed these steps to move the offsets from zookeeper to kafka:
1. Set dual commit to true, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
2. Rolling restart on all 3 brokers. All the consumers are high level
consumers with auto commit enable set to false
3. Left these settings for about an hour while data kept flowing through
some topics (not all)
4. Used the utility ./kafka-console-consumer.sh --topic __consumer_offsets
--zookeeper xxx:2181,yyy:2181,zzz:2181 --formatter
"kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
../config/consumer.properties and found that nothing is written to this
topic
5. Changed dual commit to false followed by a rolling restart. All the
consumers are high level consumers
6. Zookeeper offsets kept changing while nothing gets written to
__consumer_offsets
In order to reproduce the issue:
1. Brought down all 3 brokers and 3 nodes of zookeeper
2. Deleted all the contents of snapshot and transaction log directories of
zookeeper
3. Added myid files in snapshot directories on all zk nodes with node ids
4. Deleted all the contents of kafka log directories on all 3 brokers
5. Set dual commit to false, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
6. Brought the environment up. All the consumers are high level consumers
with auto commit enable set to false
7. Consumer offsets still got recorded on zookeeper and kept changing
while __consumer_offsets
was empty
When I did a standalone installation with single broker and used the
zookeeper that comes with kafka, the offsets got written to __consumer_offsets.
This made me ask the question about using zookeeper 3.4.6 against the one
the comes with kafka.
>You can
>also check consumer mbeans that give the KafkaCommitRate or enable
>trace logging in either the consumer or the broker's request logs to
>check if offset commit request are getting sent out to the cluster.
I will check on this
Thanks,
Kris
On Thu, Jun 11, 2015 at 7:45 AM, Joel Koshy <jj...@gmail.com> wrote:
> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> If you want to "move" offsets from zookeeper to Kafka then yes you
> need to have a phase where all consumers in your group set dual commit
> to true. If you are starting a fresh consumer group then you can
> turn off dual-commit.
>
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
>
> The zookeeper offsets won't be removed. However, are they changing?
> How are you verifying that nothing is written to this topic? If you
> are trying to consume it, then you will need to set
> exclude.internal.topics=false in your consumer properties. You can
> also check consumer mbeans that give the KafkaCommitRate or enable
> trace logging in either the consumer or the broker's request logs to
> check if offset commit request are getting sent out to the cluster.
>
> On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote:
> > I am trying to migrate the offset storage to kafka (3 brokers of version
> > 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed
> that
> > a new topic, __consumer_offsets got created.
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
> >
> > I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
> > one that comes with kafka.
> >
> > The current config consumer.properties now contains:
> >
> > offsets.storage=kafka
> > dual.commit.enabled=false
> > exclude.internal.topics=false
> >
> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> >
> > I tried both the approaches:
> >
> > 1. As listed on slide 34 of
> > http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
> > 2. By deleting the zookeeper data directories and kafka log directories.
> >
> > None of them worked.
> >
> > Thanks
> > Kris
>
>
Re: offset storage as kafka with zookeeper 3.4.6
Posted by Kris K <sq...@gmail.com>.
>If you want to "move" offsets from zookeeper to Kafka then yes you
>need to have a phase where all consumers in your group set dual commit
>to true. If you are starting a fresh consumer group then you can
>turn off dual-commit.
I followed these steps to move the offsets from zookeeper to kafka:
1. Set dual commit to true, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
2. Rolling restart on all 3 brokers. All the consumers are high level
consumers with auto commit enable set to false
3. Left these settings for about an hour while data kept flowing through
some topics (not all)
4. Used the utility ./kafka-console-consumer.sh --topic __consumer_offsets
--zookeeper xxx:2181,yyy:2181,zzz:2181 --formatter
"kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
../config/consumer.properties and found that nothing is written to this
topic
5. Changed dual commit to false followed by a rolling restart. All the
consumers are high level consumers
6. Zookeeper offsets kept changing while nothing gets written to
__consumer_offsets
In order to reproduce the issue:
1. Brought down all 3 brokers and 3 nodes of zookeeper
2. Deleted all the contents of snapshot and transaction log directories of
zookeeper
3. Added myid files in snapshot directories on all zk nodes with node ids
4. Deleted all the contents of kafka log directories on all 3 brokers
5. Set dual commit to false, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
6. Brought the environment up. All the consumers are high level consumers
with auto commit enable set to false
7. Consumer offsets still got recorded on zookeeper and kept changing
while __consumer_offsets
was empty
When I did a standalone installation with single broker and used the
zookeeper that comes with kafka, the offsets got written to __consumer_offsets.
This made me ask the question about using zookeeper 3.4.6 against the one
the comes with kafka.
>You can
>also check consumer mbeans that give the KafkaCommitRate or enable
>trace logging in either the consumer or the broker's request logs to
>check if offset commit request are getting sent out to the cluster.
I will check on this
Thanks,
Kris
On Thu, Jun 11, 2015 at 7:45 AM, Joel Koshy <jj...@gmail.com> wrote:
> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> If you want to "move" offsets from zookeeper to Kafka then yes you
> need to have a phase where all consumers in your group set dual commit
> to true. If you are starting a fresh consumer group then you can
> turn off dual-commit.
>
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
>
> The zookeeper offsets won't be removed. However, are they changing?
> How are you verifying that nothing is written to this topic? If you
> are trying to consume it, then you will need to set
> exclude.internal.topics=false in your consumer properties. You can
> also check consumer mbeans that give the KafkaCommitRate or enable
> trace logging in either the consumer or the broker's request logs to
> check if offset commit request are getting sent out to the cluster.
>
> On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote:
> > I am trying to migrate the offset storage to kafka (3 brokers of version
> > 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed
> that
> > a new topic, __consumer_offsets got created.
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
> >
> > I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
> > one that comes with kafka.
> >
> > The current config consumer.properties now contains:
> >
> > offsets.storage=kafka
> > dual.commit.enabled=false
> > exclude.internal.topics=false
> >
> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> >
> > I tried both the approaches:
> >
> > 1. As listed on slide 34 of
> > http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
> > 2. By deleting the zookeeper data directories and kafka log directories.
> >
> > None of them worked.
> >
> > Thanks
> > Kris
>
>
Re: offset storage as kafka with zookeeper 3.4.6
Posted by Joel Koshy <jj...@gmail.com>.
> Is it mandatory to use the zookeeper that comes with kafka for offset
> storage to be migrated to kafka?
If you want to "move" offsets from zookeeper to Kafka then yes you
need to have a phase where all consumers in your group set dual commit
to true. If you are starting a fresh consumer group then you can
turn off dual-commit.
> But nothing is being written to this topic, while the consumer offsets
> continue to reside on zookeeper.
The zookeeper offsets won't be removed. However, are they changing?
How are you verifying that nothing is written to this topic? If you
are trying to consume it, then you will need to set
exclude.internal.topics=false in your consumer properties. You can
also check consumer mbeans that give the KafkaCommitRate or enable
trace logging in either the consumer or the broker's request logs to
check if offset commit request are getting sent out to the cluster.
On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote:
> I am trying to migrate the offset storage to kafka (3 brokers of version
> 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that
> a new topic, __consumer_offsets got created.
> But nothing is being written to this topic, while the consumer offsets
> continue to reside on zookeeper.
>
> I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
> one that comes with kafka.
>
> The current config consumer.properties now contains:
>
> offsets.storage=kafka
> dual.commit.enabled=false
> exclude.internal.topics=false
>
> Is it mandatory to use the zookeeper that comes with kafka for offset
> storage to be migrated to kafka?
>
> I tried both the approaches:
>
> 1. As listed on slide 34 of
> http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
> 2. By deleting the zookeeper data directories and kafka log directories.
>
> None of them worked.
>
> Thanks
> Kris
Re: offset storage as kafka with zookeeper 3.4.6
Posted by Joel Koshy <jj...@gmail.com>.
> Is it mandatory to use the zookeeper that comes with kafka for offset
> storage to be migrated to kafka?
If you want to "move" offsets from zookeeper to Kafka then yes you
need to have a phase where all consumers in your group set dual commit
to true. If you are starting a fresh consumer group then you can
turn off dual-commit.
> But nothing is being written to this topic, while the consumer offsets
> continue to reside on zookeeper.
The zookeeper offsets won't be removed. However, are they changing?
How are you verifying that nothing is written to this topic? If you
are trying to consume it, then you will need to set
exclude.internal.topics=false in your consumer properties. You can
also check consumer mbeans that give the KafkaCommitRate or enable
trace logging in either the consumer or the broker's request logs to
check if offset commit request are getting sent out to the cluster.
On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote:
> I am trying to migrate the offset storage to kafka (3 brokers of version
> 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that
> a new topic, __consumer_offsets got created.
> But nothing is being written to this topic, while the consumer offsets
> continue to reside on zookeeper.
>
> I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
> one that comes with kafka.
>
> The current config consumer.properties now contains:
>
> offsets.storage=kafka
> dual.commit.enabled=false
> exclude.internal.topics=false
>
> Is it mandatory to use the zookeeper that comes with kafka for offset
> storage to be migrated to kafka?
>
> I tried both the approaches:
>
> 1. As listed on slide 34 of
> http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
> 2. By deleting the zookeeper data directories and kafka log directories.
>
> None of them worked.
>
> Thanks
> Kris