You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kiran Singh <ki...@gmail.com> on 2015/10/15 07:09:02 UTC
Difference between storing offset on Kafka and on Zookeeper server?
What are the major advantage to store Offset on kafka server instead of
zookeeper?
Please share any link for the same.
Re: Difference between storing offset on Kafka and on Zookeeper server?
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Yes, consumer group coordination is moving off of ZK in 0.9.0.0, which is
due out in November. All the new clients have zero direct dependency on ZK.
Only the brokers (and for the time being, admin and command line tools)
rely on direct access to ZK. There are plans to get a lot of admin
functionality off of ZK as well.
-Ewen
On Thu, Oct 15, 2015 at 5:50 AM, Marina <pp...@yahoo.com.invalid> wrote:
> Are there plans to move consumer group coordination off of Zk as well? And
> if so:-- what's the approximate planned release for that?-- and what
> dependencies on Zk will be left in Kafka after that?
> thanks!Marina
>
> From: Ewen Cheslack-Postava <ew...@confluent.io>
> To: "users@kafka.apache.org" <us...@kafka.apache.org>
> Sent: Thursday, October 15, 2015 2:27 AM
> Subject: Re: Difference between storing offset on Kafka and on Zookeeper
> server?
>
> There are a couple of advantages. First, scalability. Writes to Kafka are
> cheaper than writes to ZK. Kafka-based offset storage is going to be able
> to handle significantly more consumers (and scale out as needed since
> writes are spread across all partitions in the offsets topic). Second, once
> you move offsets *and* consumer group coordination off of ZK, then clients
> don't need access to ZK at all.
>
> -Ewen
>
>
>
> On Wed, Oct 14, 2015 at 10:09 PM, Kiran Singh <ki...@gmail.com>
> wrote:
>
> > What are the major advantage to store Offset on kafka server instead of
> > zookeeper?
> >
> > Please share any link for the same.
> >
>
>
>
> --
> Thanks,
> Ewen
>
>
>
>
--
Thanks,
Ewen
Re: Difference between storing offset on Kafka and on Zookeeper
server?
Posted by Marina <pp...@yahoo.com.INVALID>.
Are there plans to move consumer group coordination off of Zk as well? And if so:-- what's the approximate planned release for that?-- and what dependencies on Zk will be left in Kafka after that?
thanks!Marina
From: Ewen Cheslack-Postava <ew...@confluent.io>
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Sent: Thursday, October 15, 2015 2:27 AM
Subject: Re: Difference between storing offset on Kafka and on Zookeeper server?
There are a couple of advantages. First, scalability. Writes to Kafka are
cheaper than writes to ZK. Kafka-based offset storage is going to be able
to handle significantly more consumers (and scale out as needed since
writes are spread across all partitions in the offsets topic). Second, once
you move offsets *and* consumer group coordination off of ZK, then clients
don't need access to ZK at all.
-Ewen
On Wed, Oct 14, 2015 at 10:09 PM, Kiran Singh <ki...@gmail.com> wrote:
> What are the major advantage to store Offset on kafka server instead of
> zookeeper?
>
> Please share any link for the same.
>
--
Thanks,
Ewen
Re: Difference between storing offset on Kafka and on Zookeeper server?
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
There are a couple of advantages. First, scalability. Writes to Kafka are
cheaper than writes to ZK. Kafka-based offset storage is going to be able
to handle significantly more consumers (and scale out as needed since
writes are spread across all partitions in the offsets topic). Second, once
you move offsets *and* consumer group coordination off of ZK, then clients
don't need access to ZK at all.
-Ewen
On Wed, Oct 14, 2015 at 10:09 PM, Kiran Singh <ki...@gmail.com> wrote:
> What are the major advantage to store Offset on kafka server instead of
> zookeeper?
>
> Please share any link for the same.
>
--
Thanks,
Ewen