You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Rosenberg <jb...@squareup.com> on 2015/02/05 18:03:13 UTC
question about new consumer offset management in 0.8.2
Hi,
For 0.8.2, one of the features listed is:
- Kafka-based offset storage.
Is there documentation on this (I've heard discussion of it of course)?
Also, is it something that will be used by existing consumers when they
migrate up to 0.8.2? What is the migration process?
Thanks,
Jason
Re: question about new consumer offset management in 0.8.2
Posted by Joel Koshy <jj...@gmail.com>.
On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote:
> On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> > I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> > is kafka. Does that mean the 'default' behavior is 'false' if
> > offset.storage is 'zookeeper'? Can that be clarified in the config
> > documentation section?
> >
> > In section 5.6 where the offset managements is described, there is this:
> > "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be
> > performed using the above steps if you set offsets.storage=zookeeper."
> >
> > This implies that dual commit will work also if offsets.storage=zookeeper,
> > no? Just not by default? Perhaps there needs to be clarification there
> > (and in the config section for offsets.storage & dual.commit.enabled).
>
> Actually I think there may be a bug here if someone needs to roll back
> from Kafka-based offsets to zookeeper. Will reply tomorrow on this.
>
> >
Never mind - I think we are fine here. The scenario I was thinking
about is the following:
- If there are three consumer instances c0, c1, c2 consuming
partitions pX, pY, ... and are committing offsets to Kafka and you
want to migrate to zookeeper
- Do a rolling bounce to turn on dual-commit (and keep offset.storage
= kafka)
- Do another rolling bounce to set offset.storage to zookeeper:
- Say, you bounce c0 to commit offsets to zk and it comes back up
and then owns pX. It begins to commit offsets for pX to zookeeper
only.
- You then bounce c1; after it goes down due to our partition
assignment strategy say pX is now assigned to c2 (which has not
yet been bounced).
- c2 uses offset.storage=kafka so would fetch a potentially stale
offset for pX which would be an issue.
- So we explicitly handle this case - if dual.commit is turned on
and offset.storage is kafka, then the broker fetches offsets from
both Kafka and ZooKeeper and selects the maximum of the two.
Let me know if you see any holes in the above.
dual.commit is confusing and would have been (slightly) less confusing if it
was called offset.migration.in.progress or something similar. Still, I think
we can document the process carefully and state clearly that it is
intended for use during migration/roll-back only.
Re: question about new consumer offset management in 0.8.2
Posted by Joel Koshy <jj...@gmail.com>.
On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> is kafka. Does that mean the 'default' behavior is 'false' if
> offset.storage is 'zookeeper'? Can that be clarified in the config
> documentation section?
>
> In section 5.6 where the offset managements is described, there is this:
> "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be
> performed using the above steps if you set offsets.storage=zookeeper."
>
> This implies that dual commit will work also if offsets.storage=zookeeper,
> no? Just not by default? Perhaps there needs to be clarification there
> (and in the config section for offsets.storage & dual.commit.enabled).
Actually I think there may be a bug here if someone needs to roll back
from Kafka-based offsets to zookeeper. Will reply tomorrow on this.
>
> The doc in section 5.6 is probably in need of editing, it looks like it in
> places assumes zookeeper offset storage, and has some repeated sentences,
> etc.
Will do.
>
> Finally, why is section 5.6 titled "Distribution"? Seems to be a grab-bag
> of mostly consumer related topics?
Yes this was prior structure that can be improved.
> >
> > > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jj...@gmail.com> wrote:
> > >
> > > > This is documented in the official docs:
> > > > http://kafka.apache.org/documentation.html#distributionimpl
> > > >
> > > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > > > > What are the defaults for those settings (I assume it will be to
> > continue
> > > > > using only zookeeper by default)?
> > > > >
> > > > > Also, if I have a cluster of consumers sharing the same groupId, and
> > I
> > > > > update them via a rolling release, will it be a problem during the
> > > > rolling
> > > > > restart if there is inconsistency in the settings for a short time?
> > Or
> > > > is
> > > > > it required that the entire cluster be stopped, then update configs,
> > then
> > > > > restart all nodes?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshapira@cloudera.com
> > >
> > > > wrote:
> > > > >
> > > > > > Thanks Jon. I updated the FAQ with your procedure:
> > > > > >
> > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > > > > ?
> > > > > >
> > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > > > > jbringhurst@linkedin.com.invalid> wrote:
> > > > > >
> > > > > > > There should probably be a wiki page started for this so we have
> > the
> > > > > > > details in one place. The same question was asked on Freenode
> > IRC a
> > > > few
> > > > > > > minutes ago. :)
> > > > > > >
> > > > > > > A summary of the migration procedure is:
> > > > > > >
> > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and
> > restart
> > > > > > > (Commit offsets to Zookeeper and Kafka).
> > > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and
> > > > restart
> > > > > > > (Commit offsets to Kafka only).
> > > > > > >
> > > > > > > -Jon
> > > > > > >
> > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > For 0.8.2, one of the features listed is:
> > > > > > > > - Kafka-based offset storage.
> > > > > > > >
> > > > > > > > Is there documentation on this (I've heard discussion of it of
> > > > course)?
> > > > > > > >
> > > > > > > > Also, is it something that will be used by existing consumers
> > when
> > > > they
> > > > > > > > migrate up to 0.8.2? What is the migration process?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> >
> > --
> > Joel
> >
Re: question about new consumer offset management in 0.8.2
Posted by Jason Rosenberg <jb...@squareup.com>.
On Thu, Feb 5, 2015 at 9:52 PM, Joel Koshy <jj...@gmail.com> wrote:
> > Ok, so it looks like the default settings are:
> > offset.storage = zookeeper
> > dual.commit.enabled = true
> > The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
> > state) that it will only apply if offset.storage = kafka. Is that right?
> > (I'm guessing not)
>
> dual.commit.enabled defaults to true only if offset.storage is kafka.
> As you noted, it only applies if offset.storage = kafka is primarily
> intended for migration.
>
I'm not sure what you mean by 'default' behavior 'only if' offset.storage
is kafka. Does that mean the 'default' behavior is 'false' if
offset.storage is 'zookeeper'? Can that be clarified in the config
documentation section?
In section 5.6 where the offset managements is described, there is this:
"A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be
performed using the above steps if you set offsets.storage=zookeeper."
This implies that dual commit will work also if offsets.storage=zookeeper,
no? Just not by default? Perhaps there needs to be clarification there
(and in the config section for offsets.storage & dual.commit.enabled).
The doc in section 5.6 is probably in need of editing, it looks like it in
places assumes zookeeper offset storage, and has some repeated sentences,
etc.
Finally, why is section 5.6 titled "Distribution"? Seems to be a grab-bag
of mostly consumer related topics?
> > It seems to me less than ideal to have the default behavior to have
> > dual.commit.enabled = true, since this seems like a performance hit, no?
>
> To some degree yes, but it is relatively cheap.
>
And as you pointed out (I think) it's not even an issue because it won't be
dual committing initially, by default (while offsets.storage = zookeeper),
right?
> > Also, I assume the __consumer_offsets topic will be set to have an
> infinite
> > retention policy internally, is that right? So that currently committed
> > offsets for a given consumer group won't be lost?
>
> It uses the "compaction" retention policy - so the topic won't grow
> unbounded.
>
Nice
> Thanks,
>
> Joel
>
> > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> > > This is documented in the official docs:
> > > http://kafka.apache.org/documentation.html#distributionimpl
> > >
> > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > > > What are the defaults for those settings (I assume it will be to
> continue
> > > > using only zookeeper by default)?
> > > >
> > > > Also, if I have a cluster of consumers sharing the same groupId, and
> I
> > > > update them via a rolling release, will it be a problem during the
> > > rolling
> > > > restart if there is inconsistency in the settings for a short time?
> Or
> > > is
> > > > it required that the entire cluster be stopped, then update configs,
> then
> > > > restart all nodes?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > > wrote:
> > > >
> > > > > Thanks Jon. I updated the FAQ with your procedure:
> > > > >
> > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > > > ?
> > > > >
> > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > > > jbringhurst@linkedin.com.invalid> wrote:
> > > > >
> > > > > > There should probably be a wiki page started for this so we have
> the
> > > > > > details in one place. The same question was asked on Freenode
> IRC a
> > > few
> > > > > > minutes ago. :)
> > > > > >
> > > > > > A summary of the migration procedure is:
> > > > > >
> > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and
> restart
> > > > > > (Commit offsets to Zookeeper and Kafka).
> > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and
> > > restart
> > > > > > (Commit offsets to Kafka only).
> > > > > >
> > > > > > -Jon
> > > > > >
> > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com>
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > For 0.8.2, one of the features listed is:
> > > > > > > - Kafka-based offset storage.
> > > > > > >
> > > > > > > Is there documentation on this (I've heard discussion of it of
> > > course)?
> > > > > > >
> > > > > > > Also, is it something that will be used by existing consumers
> when
> > > they
> > > > > > > migrate up to 0.8.2? What is the migration process?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jason
> > > > > >
> > > > > >
> > > > >
> > >
> > >
>
> --
> Joel
>
Re: question about new consumer offset management in 0.8.2
Posted by Joel Koshy <jj...@gmail.com>.
> Ok, so it looks like the default settings are:
> offset.storage = zookeeper
> dual.commit.enabled = true
> The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
> state) that it will only apply if offset.storage = kafka. Is that right?
> (I'm guessing not)
dual.commit.enabled defaults to true only if offset.storage is kafka.
As you noted, it only applies if offset.storage = kafka is primarily
intended for migration.
> It seems to me less than ideal to have the default behavior to have
> dual.commit.enabled = true, since this seems like a performance hit, no?
To some degree yes, but it is relatively cheap.
> I'd think you'd only want this during a planned migration.
Yes.
>
> Also, I assume it's desirable to switch to using 'kafka' for offset
> storage, for performance reasons? Will it better handle a larger number of
> topics?
Yes.
> Also, I assume the __consumer_offsets topic will be set to have an infinite
> retention policy internally, is that right? So that currently committed
> offsets for a given consumer group won't be lost?
It uses the "compaction" retention policy - so the topic won't grow
unbounded. Compaction will basically dedupe on the inactive segments
of the topic - so in effect it will only maintain the last committed
offset for a given group-topic-partition 3-tuple (plus a few - since
it runs only when a certain dirtiness threshold has been met). The
compaction policy is also documented on the site.
Thanks,
Joel
> On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > This is documented in the official docs:
> > http://kafka.apache.org/documentation.html#distributionimpl
> >
> > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > > What are the defaults for those settings (I assume it will be to continue
> > > using only zookeeper by default)?
> > >
> > > Also, if I have a cluster of consumers sharing the same groupId, and I
> > > update them via a rolling release, will it be a problem during the
> > rolling
> > > restart if there is inconsistency in the settings for a short time? Or
> > is
> > > it required that the entire cluster be stopped, then update configs, then
> > > restart all nodes?
> > >
> > > Jason
> > >
> > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> > >
> > > > Thanks Jon. I updated the FAQ with your procedure:
> > > >
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > > ?
> > > >
> > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > > jbringhurst@linkedin.com.invalid> wrote:
> > > >
> > > > > There should probably be a wiki page started for this so we have the
> > > > > details in one place. The same question was asked on Freenode IRC a
> > few
> > > > > minutes ago. :)
> > > > >
> > > > > A summary of the migration procedure is:
> > > > >
> > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> > > > > (Commit offsets to Zookeeper and Kafka).
> > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and
> > restart
> > > > > (Commit offsets to Kafka only).
> > > > >
> > > > > -Jon
> > > > >
> > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > For 0.8.2, one of the features listed is:
> > > > > > - Kafka-based offset storage.
> > > > > >
> > > > > > Is there documentation on this (I've heard discussion of it of
> > course)?
> > > > > >
> > > > > > Also, is it something that will be used by existing consumers when
> > they
> > > > > > migrate up to 0.8.2? What is the migration process?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jason
> > > > >
> > > > >
> > > >
> >
> >
--
Joel
Re: question about new consumer offset management in 0.8.2
Posted by Jason Rosenberg <jb...@squareup.com>.
Ok, so it looks like the default settings are:
offset.storage = zookeeper
dual.commit.enabled = true
The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
state) that it will only apply if offset.storage = kafka. Is that right?
(I'm guessing not)
"*If you are using "kafka"* as offsets.storage, you can dual commit offsets
to ZooKeeper (in addition to Kafka)."
It seems to me less than ideal to have the default behavior to have
dual.commit.enabled = true, since this seems like a performance hit, no?
I'd think you'd only want this during a planned migration.
Also, I assume it's desirable to switch to using 'kafka' for offset
storage, for performance reasons? Will it better handle a larger number of
topics?
Also, I assume the __consumer_offsets topic will be set to have an infinite
retention policy internally, is that right? So that currently committed
offsets for a given consumer group won't be lost?
Jason
On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jj...@gmail.com> wrote:
> This is documented in the official docs:
> http://kafka.apache.org/documentation.html#distributionimpl
>
> On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > What are the defaults for those settings (I assume it will be to continue
> > using only zookeeper by default)?
> >
> > Also, if I have a cluster of consumers sharing the same groupId, and I
> > update them via a rolling release, will it be a problem during the
> rolling
> > restart if there is inconsistency in the settings for a short time? Or
> is
> > it required that the entire cluster be stopped, then update configs, then
> > restart all nodes?
> >
> > Jason
> >
> > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >
> > > Thanks Jon. I updated the FAQ with your procedure:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > ?
> > >
> > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > jbringhurst@linkedin.com.invalid> wrote:
> > >
> > > > There should probably be a wiki page started for this so we have the
> > > > details in one place. The same question was asked on Freenode IRC a
> few
> > > > minutes ago. :)
> > > >
> > > > A summary of the migration procedure is:
> > > >
> > > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> > > > (Commit offsets to Zookeeper and Kafka).
> > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and
> restart
> > > > (Commit offsets to Kafka only).
> > > >
> > > > -Jon
> > > >
> > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > For 0.8.2, one of the features listed is:
> > > > > - Kafka-based offset storage.
> > > > >
> > > > > Is there documentation on this (I've heard discussion of it of
> course)?
> > > > >
> > > > > Also, is it something that will be used by existing consumers when
> they
> > > > > migrate up to 0.8.2? What is the migration process?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jason
> > > >
> > > >
> > >
>
>
Re: question about new consumer offset management in 0.8.2
Posted by Joel Koshy <jj...@gmail.com>.
This is documented in the official docs:
http://kafka.apache.org/documentation.html#distributionimpl
On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> What are the defaults for those settings (I assume it will be to continue
> using only zookeeper by default)?
>
> Also, if I have a cluster of consumers sharing the same groupId, and I
> update them via a rolling release, will it be a problem during the rolling
> restart if there is inconsistency in the settings for a short time? Or is
> it required that the entire cluster be stopped, then update configs, then
> restart all nodes?
>
> Jason
>
> On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
> > Thanks Jon. I updated the FAQ with your procedure:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > ?
> >
> > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > jbringhurst@linkedin.com.invalid> wrote:
> >
> > > There should probably be a wiki page started for this so we have the
> > > details in one place. The same question was asked on Freenode IRC a few
> > > minutes ago. :)
> > >
> > > A summary of the migration procedure is:
> > >
> > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> > > (Commit offsets to Zookeeper and Kafka).
> > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart
> > > (Commit offsets to Kafka only).
> > >
> > > -Jon
> > >
> > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > For 0.8.2, one of the features listed is:
> > > > - Kafka-based offset storage.
> > > >
> > > > Is there documentation on this (I've heard discussion of it of course)?
> > > >
> > > > Also, is it something that will be used by existing consumers when they
> > > > migrate up to 0.8.2? What is the migration process?
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > >
> > >
> >
Re: question about new consumer offset management in 0.8.2
Posted by Jason Rosenberg <jb...@squareup.com>.
What are the defaults for those settings (I assume it will be to continue
using only zookeeper by default)?
Also, if I have a cluster of consumers sharing the same groupId, and I
update them via a rolling release, will it be a problem during the rolling
restart if there is inconsistency in the settings for a short time? Or is
it required that the entire cluster be stopped, then update configs, then
restart all nodes?
Jason
On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> Thanks Jon. I updated the FAQ with your procedure:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> ?
>
> On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> jbringhurst@linkedin.com.invalid> wrote:
>
> > There should probably be a wiki page started for this so we have the
> > details in one place. The same question was asked on Freenode IRC a few
> > minutes ago. :)
> >
> > A summary of the migration procedure is:
> >
> > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> > (Commit offsets to Zookeeper and Kafka).
> > 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart
> > (Commit offsets to Kafka only).
> >
> > -Jon
> >
> > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> >
> > > Hi,
> > >
> > > For 0.8.2, one of the features listed is:
> > > - Kafka-based offset storage.
> > >
> > > Is there documentation on this (I've heard discussion of it of course)?
> > >
> > > Also, is it something that will be used by existing consumers when they
> > > migrate up to 0.8.2? What is the migration process?
> > >
> > > Thanks,
> > >
> > > Jason
> >
> >
>
Re: question about new consumer offset management in 0.8.2
Posted by Gwen Shapira <gs...@cloudera.com>.
Thanks Jon. I updated the FAQ with your procedure:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
?
On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
jbringhurst@linkedin.com.invalid> wrote:
> There should probably be a wiki page started for this so we have the
> details in one place. The same question was asked on Freenode IRC a few
> minutes ago. :)
>
> A summary of the migration procedure is:
>
> 1) Upgrade your brokers and set dual.commit.enabled=false and
> offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> (Commit offsets to Zookeeper and Kafka).
> 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart
> (Commit offsets to Kafka only).
>
> -Jon
>
> On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > Hi,
> >
> > For 0.8.2, one of the features listed is:
> > - Kafka-based offset storage.
> >
> > Is there documentation on this (I've heard discussion of it of course)?
> >
> > Also, is it something that will be used by existing consumers when they
> > migrate up to 0.8.2? What is the migration process?
> >
> > Thanks,
> >
> > Jason
>
>
Re: question about new consumer offset management in 0.8.2
Posted by Surendranauth Hiraman <su...@velos.io>.
This is what I've found so far.
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
The high-level consumer just worked for me by setting offsets.storage =
kafka.
Scroll down to the offsets.* config params.
http://kafka.apache.org/documentation.html#consumerconfigs
-Suren
On Thu, Feb 5, 2015 at 12:16 PM, Jon Bringhurst <
jbringhurst@linkedin.com.invalid> wrote:
> There should probably be a wiki page started for this so we have the
> details in one place. The same question was asked on Freenode IRC a few
> minutes ago. :)
>
> A summary of the migration procedure is:
>
> 1) Upgrade your brokers and set dual.commit.enabled=false and
> offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> (Commit offsets to Zookeeper and Kafka).
> 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart
> (Commit offsets to Kafka only).
>
> -Jon
>
> On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > Hi,
> >
> > For 0.8.2, one of the features listed is:
> > - Kafka-based offset storage.
> >
> > Is there documentation on this (I've heard discussion of it of course)?
> >
> > Also, is it something that will be used by existing consumers when they
> > migrate up to 0.8.2? What is the migration process?
> >
> > Thanks,
> >
> > Jason
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
54 West 40th Street, 3RD FLOOR
NEW YORK, NY 10018
T: @suren_h
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: question about new consumer offset management in 0.8.2
Posted by Jon Bringhurst <jb...@linkedin.com.INVALID>.
There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :)
A summary of the migration procedure is:
1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka).
3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only).
-Jon
On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> Hi,
>
> For 0.8.2, one of the features listed is:
> - Kafka-based offset storage.
>
> Is there documentation on this (I've heard discussion of it of course)?
>
> Also, is it something that will be used by existing consumers when they
> migrate up to 0.8.2? What is the migration process?
>
> Thanks,
>
> Jason