You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by S Ahmed <sa...@gmail.com> on 2014/06/04 19:00:02 UTC
are consumer offsets stored in a log?
I swear I read that Jay Kreps wrote somewhere that consumers now write
their offsets in a logfile (not in zookeeper).
Is this true or did I misread? Sorry I can't find the article I was
reading.
Re: are consumer offsets stored in a log?
Posted by Neha Narkhede <ne...@gmail.com>.
Consumers in a future release (0.8.2) or trunk will write offsets to a
Kafka topic, not zookeeper.
Thanks,
Neha
On Jun 4, 2014 10:00 AM, "S Ahmed" <sa...@gmail.com> wrote:
> I swear I read that Jay Kreps wrote somewhere that consumers now write
> their offsets in a logfile (not in zookeeper).
>
> Is this true or did I misread? Sorry I can't find the article I was
> reading.
>
Re: are consumer offsets stored in a log?
Posted by Dennis Haller <dh...@talemetry.com>.
This will force a rewrite of those monitoring tools and UI tools that read
offsets from Zookeeper in order directly to get lag information for
reporting on consumer clients. It seems a good thing to know this is coming
down the pipe.
Dennis
On Wed, Jun 4, 2014 at 6:50 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> At LinkedIn, we are just beginning to move our applications to Kafka based
> offset management. I'm not sure if Todd/Clark/Joel have any numbers to
> share right away, but depending on the number of consumers you have and the
> number of topics those consumers pull, it will likely make a sizable impact
> on the performance of your zookeeper cluster. It will not, however,
> influence the decision of sizing the zookeeper cluster since zookeeper
> writes do not scale with the number of servers in a zookeeper cluster. In
> general, it is recommended to keep a zookeeper cluster small, of the size
> of 3 or 5 servers. i.e. whether or not your consumers checkpoint offsets to
> zookeeper or Kafka. It is true that the writes will be faster on a 3 node
> zookeeper cluster vs a 5 node zookeeper cluster.
>
> Thanks
> Neha
>
>
> On Wed, Jun 4, 2014 at 11:15 AM, S Ahmed <sa...@gmail.com> wrote:
>
> > Very nice.
> >
> > Do you guys have any stats on what kind of load was reduced on ZK? Just
> > trying to understand if this changes the type of servers required to host
> > ZK.
> >
> >
> >
> >
> > On Wed, Jun 4, 2014 at 1:10 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Yes, we are migrating the offset management from ZK to the broker as a
> > > special log.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
> > >
> > > The code is in trunk, and it is running in production at LinkedIn now.
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sa...@gmail.com> wrote:
> > >
> > > > I swear I read that Jay Kreps wrote somewhere that consumers now
> write
> > > > their offsets in a logfile (not in zookeeper).
> > > >
> > > > Is this true or did I misread? Sorry I can't find the article I was
> > > > reading.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: are consumer offsets stored in a log?
Posted by Neha Narkhede <ne...@gmail.com>.
At LinkedIn, we are just beginning to move our applications to Kafka based
offset management. I'm not sure if Todd/Clark/Joel have any numbers to
share right away, but depending on the number of consumers you have and the
number of topics those consumers pull, it will likely make a sizable impact
on the performance of your zookeeper cluster. It will not, however,
influence the decision of sizing the zookeeper cluster since zookeeper
writes do not scale with the number of servers in a zookeeper cluster. In
general, it is recommended to keep a zookeeper cluster small, of the size
of 3 or 5 servers. i.e. whether or not your consumers checkpoint offsets to
zookeeper or Kafka. It is true that the writes will be faster on a 3 node
zookeeper cluster vs a 5 node zookeeper cluster.
Thanks
Neha
On Wed, Jun 4, 2014 at 11:15 AM, S Ahmed <sa...@gmail.com> wrote:
> Very nice.
>
> Do you guys have any stats on what kind of load was reduced on ZK? Just
> trying to understand if this changes the type of servers required to host
> ZK.
>
>
>
>
> On Wed, Jun 4, 2014 at 1:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Yes, we are migrating the offset management from ZK to the broker as a
> > special log.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
> >
> > The code is in trunk, and it is running in production at LinkedIn now.
> >
> > Guozhang
> >
> >
> > On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sa...@gmail.com> wrote:
> >
> > > I swear I read that Jay Kreps wrote somewhere that consumers now write
> > > their offsets in a logfile (not in zookeeper).
> > >
> > > Is this true or did I misread? Sorry I can't find the article I was
> > > reading.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: are consumer offsets stored in a log?
Posted by S Ahmed <sa...@gmail.com>.
Very nice.
Do you guys have any stats on what kind of load was reduced on ZK? Just
trying to understand if this changes the type of servers required to host
ZK.
On Wed, Jun 4, 2014 at 1:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Yes, we are migrating the offset management from ZK to the broker as a
> special log.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
>
> The code is in trunk, and it is running in production at LinkedIn now.
>
> Guozhang
>
>
> On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sa...@gmail.com> wrote:
>
> > I swear I read that Jay Kreps wrote somewhere that consumers now write
> > their offsets in a logfile (not in zookeeper).
> >
> > Is this true or did I misread? Sorry I can't find the article I was
> > reading.
> >
>
>
>
> --
> -- Guozhang
>
Re: are consumer offsets stored in a log?
Posted by Guozhang Wang <wa...@gmail.com>.
Yes, we are migrating the offset management from ZK to the broker as a
special log.
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
The code is in trunk, and it is running in production at LinkedIn now.
Guozhang
On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sa...@gmail.com> wrote:
> I swear I read that Jay Kreps wrote somewhere that consumers now write
> their offsets in a logfile (not in zookeeper).
>
> Is this true or did I misread? Sorry I can't find the article I was
> reading.
>
--
-- Guozhang