You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Andrii Biletskyi <an...@stealth.ly> on 2015/02/16 18:18:08 UTC
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Hi all,
I've added some details to Proposed Changes paragraph.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
Please let me know if something else require description / details.
Thanks,
Andrii
On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps <ja...@confluent.io> wrote:
> Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come across
> the wrong way. But the hope I really have for these is that they are
> complete enough that even highly involved users can see and understand the
> change, motivation, etc. I think that will do a ton to help extend the
> community.
>
> -Jay
>
> On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > There is still some to-dos to be done in
> > https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
> > https://reviews.apache.org/r/30126/diff/ once that is in.
> >
> > We can get more written up on it, will do.
> >
> > On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Hey Joe,
> > >
> > > Can you fill in this KIP? The purpose of these KIPs is to give a full
> > > overview of the feature, how it will work, be implemented, the
> > > considerations involved, etc. There is only like one sentence on this
> > which
> > > isn't enough for anyone to know what you are thinking.
> > >
> > > Moving off of configs to something that I'm guessing would be
> > > Zookeeper-based (?) is a massive change so we really need to describe
> > this
> > > in a way that can be widely circulated.
> > >
> > > I actually think this would be a good idea. But there are a ton of
> > > advantages to good old fashioned text files in terms of config
> management
> > > and change control. And trying to support both may or may not be
> better.
> > >
> > > -Jay
> > >
> > >
> > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > wrote:
> > >
> > > > Created a KIP
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > >
> > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Neha Narkhede <ne...@confluent.io>.
Andrii,
Thanks for updating the KIP. Few comments -
Would you mind updating the Compatibility, Migration Plan section as well?
We should discuss the new config store options - I'm not so sure that
zookeeper is such a slam dunk choice for storing the configs.
1. Firstly, we have seen quite a few problems with zkClient which is
currently the client we use to talk to ZooKeeper. In our observation, the
extra layer of indirection it provides seemed to be useful when we were
starting off but the more production experience we got, the more problems
popped up. These include missed watches
<https://issues.apache.org/jira/browse/KAFKA-1155> in some cases, no
timeouts <https://issues.apache.org/jira/browse/KAFKA-1907>, limited
APIs etc. This makes it much trickier to depend on ZooKeeper for
important state information that needs to propagate in the cluster. We may
have to investigate writing our own utility for interacting
with ZooKeeper but even then it is not going to get much easier since we
still have to solve the corner case issues we ran into with zkClient.
2. Secondly, this adds yet another piece of important state information
we store in ZooKeeper assuming that zookeeper is the easiest alternative
for key-value storage. However, we have gotten burnt by this in the past
and still are. For example, admin commands like preferred replica election
and partition reassignment spend quite some time updating the progress of
the admin operation in ZooKeeper and hence slowing down the operation
since ZooKeeper is not optimized for sudden spikes in write throughput.
Also, I don't know if the argument that the information is necessarily
safer in ZooKeeper is quite convincing, especially, with the successful
move of consumer offsets from ZooKeeper to Kafka. It may not be a bad
option to consider storing configs in a special durable Kafka topic. I'm
not necessarily advocating this but would definitely like to discuss it.
Before we move forward on adding more important state information in
ZooKeeper, I think we should make an effort at reducing some technical debt
we have accumulated and clean that up first.
On Mon, Feb 16, 2015 at 9:18 AM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Hi all,
>
> I've added some details to Proposed Changes paragraph.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>
> Please let me know if something else require description / details.
>
> Thanks,
> Andrii
>
> On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come
> across
> > the wrong way. But the hope I really have for these is that they are
> > complete enough that even highly involved users can see and understand
> the
> > change, motivation, etc. I think that will do a ton to help extend the
> > community.
> >
> > -Jay
> >
> > On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >
> > > There is still some to-dos to be done in
> > > https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
> > > https://reviews.apache.org/r/30126/diff/ once that is in.
> > >
> > > We can get more written up on it, will do.
> > >
> > > On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >
> > > > Hey Joe,
> > > >
> > > > Can you fill in this KIP? The purpose of these KIPs is to give a full
> > > > overview of the feature, how it will work, be implemented, the
> > > > considerations involved, etc. There is only like one sentence on this
> > > which
> > > > isn't enough for anyone to know what you are thinking.
> > > >
> > > > Moving off of configs to something that I'm guessing would be
> > > > Zookeeper-based (?) is a massive change so we really need to describe
> > > this
> > > > in a way that can be widely circulated.
> > > >
> > > > I actually think this would be a good idea. But there are a ton of
> > > > advantages to good old fashioned text files in terms of config
> > management
> > > > and change control. And trying to support both may or may not be
> > better.
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > > wrote:
> > > >
> > > > > Created a KIP
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop
> >
> > > > > ********************************************/
> > > > >
> > > >
> > >
> >
>
--
Thanks,
Neha