You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mike Marzo <pr...@gmail.com> on 2014/09/10 17:20:12 UTC
High level consumer with separate zk
Is it possible for the high level consumer to use a different zk cluster
than the cluster that manages broker leader electivity?
The high level consumer adds a lot of value but I don't like the idea that
bad user code could pound the core zk and effectively hurt the kafka
brokers
mike marzo
908 209-4484
Re: High level consumer with separate zk
Posted by Jun Rao <ju...@gmail.com>.
Storing consumer offsets in the broker will add a bit overhead to the
broker, but shouldn't be too bad. Compared with traditional messaging
systems, Kafka delivers messages to consumers in order and therefore only
needs to keep track of a single last consumed offset per consumer client
per topic partition. Also, Kafka can scale horizontally and the tracked
consumer offsets can be spread over multiple brokers.
Thanks,
Jun
On Wed, Sep 10, 2014 at 8:31 AM, Mike Marzo <pr...@gmail.com>
wrote:
> But that could potentially impact the brokers directly. Which is my core
> concern.
> Is there any idea of the perf impact of having brokers be client aware.
> This moves kafka more toward the traditional mom style messaging broker...
>
> mike marzo
> 908 209-4484
> On Sep 10, 2014 11:24 AM, "Jonathan Weeks" <jo...@gmail.com>
> wrote:
>
> > When 0.8.2 arrives in the near future, consumer offsets will be stored by
> > the brokers, and thus that workload will not be impacting ZK.
> >
> > Best Regards,
> >
> > -Jonathan
> >
> >
> > On Sep 10, 2014, at 8:20 AM, Mike Marzo <pr...@gmail.com>
> > wrote:
> >
> > > Is it possible for the high level consumer to use a different zk
> cluster
> > > than the cluster that manages broker leader electivity?
> > > The high level consumer adds a lot of value but I don't like the idea
> > that
> > > bad user code could pound the core zk and effectively hurt the kafka
> > > brokers
> > > mike marzo
> > > 908 209-4484
> >
> >
>
Re: High level consumer with separate zk
Posted by Mike Marzo <pr...@gmail.com>.
But that could potentially impact the brokers directly. Which is my core
concern.
Is there any idea of the perf impact of having brokers be client aware.
This moves kafka more toward the traditional mom style messaging broker...
mike marzo
908 209-4484
On Sep 10, 2014 11:24 AM, "Jonathan Weeks" <jo...@gmail.com> wrote:
> When 0.8.2 arrives in the near future, consumer offsets will be stored by
> the brokers, and thus that workload will not be impacting ZK.
>
> Best Regards,
>
> -Jonathan
>
>
> On Sep 10, 2014, at 8:20 AM, Mike Marzo <pr...@gmail.com>
> wrote:
>
> > Is it possible for the high level consumer to use a different zk cluster
> > than the cluster that manages broker leader electivity?
> > The high level consumer adds a lot of value but I don't like the idea
> that
> > bad user code could pound the core zk and effectively hurt the kafka
> > brokers
> > mike marzo
> > 908 209-4484
>
>
Re: High level consumer with separate zk
Posted by Jonathan Weeks <jo...@gmail.com>.
When 0.8.2 arrives in the near future, consumer offsets will be stored by the brokers, and thus that workload will not be impacting ZK.
Best Regards,
-Jonathan
On Sep 10, 2014, at 8:20 AM, Mike Marzo <pr...@gmail.com> wrote:
> Is it possible for the high level consumer to use a different zk cluster
> than the cluster that manages broker leader electivity?
> The high level consumer adds a lot of value but I don't like the idea that
> bad user code could pound the core zk and effectively hurt the kafka
> brokers
> mike marzo
> 908 209-4484