You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Joe Stein <jo...@stealth.ly> on 2015/01/22 07:34:37 UTC
[DISCUSS] KIP-5 - Broker Configuration Management
Created a KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
JIRA https://issues.apache.org/jira/browse/KAFKA-1786
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Neha Narkhede <ne...@confluent.io>.
Andrii,
Thanks for updating the KIP. Few comments -
Would you mind updating the Compatibility, Migration Plan section as well?
We should discuss the new config store options - I'm not so sure that
zookeeper is such a slam dunk choice for storing the configs.
1. Firstly, we have seen quite a few problems with zkClient which is
currently the client we use to talk to ZooKeeper. In our observation, the
extra layer of indirection it provides seemed to be useful when we were
starting off but the more production experience we got, the more problems
popped up. These include missed watches
<https://issues.apache.org/jira/browse/KAFKA-1155> in some cases, no
timeouts <https://issues.apache.org/jira/browse/KAFKA-1907>, limited
APIs etc. This makes it much trickier to depend on ZooKeeper for
important state information that needs to propagate in the cluster. We may
have to investigate writing our own utility for interacting
with ZooKeeper but even then it is not going to get much easier since we
still have to solve the corner case issues we ran into with zkClient.
2. Secondly, this adds yet another piece of important state information
we store in ZooKeeper assuming that zookeeper is the easiest alternative
for key-value storage. However, we have gotten burnt by this in the past
and still are. For example, admin commands like preferred replica election
and partition reassignment spend quite some time updating the progress of
the admin operation in ZooKeeper and hence slowing down the operation
since ZooKeeper is not optimized for sudden spikes in write throughput.
Also, I don't know if the argument that the information is necessarily
safer in ZooKeeper is quite convincing, especially, with the successful
move of consumer offsets from ZooKeeper to Kafka. It may not be a bad
option to consider storing configs in a special durable Kafka topic. I'm
not necessarily advocating this but would definitely like to discuss it.
Before we move forward on adding more important state information in
ZooKeeper, I think we should make an effort at reducing some technical debt
we have accumulated and clean that up first.
On Mon, Feb 16, 2015 at 9:18 AM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Hi all,
>
> I've added some details to Proposed Changes paragraph.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>
> Please let me know if something else require description / details.
>
> Thanks,
> Andrii
>
> On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come
> across
> > the wrong way. But the hope I really have for these is that they are
> > complete enough that even highly involved users can see and understand
> the
> > change, motivation, etc. I think that will do a ton to help extend the
> > community.
> >
> > -Jay
> >
> > On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >
> > > There is still some to-dos to be done in
> > > https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
> > > https://reviews.apache.org/r/30126/diff/ once that is in.
> > >
> > > We can get more written up on it, will do.
> > >
> > > On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >
> > > > Hey Joe,
> > > >
> > > > Can you fill in this KIP? The purpose of these KIPs is to give a full
> > > > overview of the feature, how it will work, be implemented, the
> > > > considerations involved, etc. There is only like one sentence on this
> > > which
> > > > isn't enough for anyone to know what you are thinking.
> > > >
> > > > Moving off of configs to something that I'm guessing would be
> > > > Zookeeper-based (?) is a massive change so we really need to describe
> > > this
> > > > in a way that can be widely circulated.
> > > >
> > > > I actually think this would be a good idea. But there are a ton of
> > > > advantages to good old fashioned text files in terms of config
> > management
> > > > and change control. And trying to support both may or may not be
> > better.
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > > wrote:
> > > >
> > > > > Created a KIP
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop
> >
> > > > > ********************************************/
> > > > >
> > > >
> > >
> >
>
--
Thanks,
Neha
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Andrii Biletskyi <an...@stealth.ly>.
Hi all,
I've added some details to Proposed Changes paragraph.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
Please let me know if something else require description / details.
Thanks,
Andrii
On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps <ja...@confluent.io> wrote:
> Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come across
> the wrong way. But the hope I really have for these is that they are
> complete enough that even highly involved users can see and understand the
> change, motivation, etc. I think that will do a ton to help extend the
> community.
>
> -Jay
>
> On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > There is still some to-dos to be done in
> > https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
> > https://reviews.apache.org/r/30126/diff/ once that is in.
> >
> > We can get more written up on it, will do.
> >
> > On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Hey Joe,
> > >
> > > Can you fill in this KIP? The purpose of these KIPs is to give a full
> > > overview of the feature, how it will work, be implemented, the
> > > considerations involved, etc. There is only like one sentence on this
> > which
> > > isn't enough for anyone to know what you are thinking.
> > >
> > > Moving off of configs to something that I'm guessing would be
> > > Zookeeper-based (?) is a massive change so we really need to describe
> > this
> > > in a way that can be widely circulated.
> > >
> > > I actually think this would be a good idea. But there are a ton of
> > > advantages to good old fashioned text files in terms of config
> management
> > > and change control. And trying to support both may or may not be
> better.
> > >
> > > -Jay
> > >
> > >
> > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > wrote:
> > >
> > > > Created a KIP
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > >
> > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Jay Kreps <ja...@confluent.io>.
Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come across
the wrong way. But the hope I really have for these is that they are
complete enough that even highly involved users can see and understand the
change, motivation, etc. I think that will do a ton to help extend the
community.
-Jay
On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <jo...@stealth.ly> wrote:
> There is still some to-dos to be done in
> https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
> https://reviews.apache.org/r/30126/diff/ once that is in.
>
> We can get more written up on it, will do.
>
> On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hey Joe,
> >
> > Can you fill in this KIP? The purpose of these KIPs is to give a full
> > overview of the feature, how it will work, be implemented, the
> > considerations involved, etc. There is only like one sentence on this
> which
> > isn't enough for anyone to know what you are thinking.
> >
> > Moving off of configs to something that I'm guessing would be
> > Zookeeper-based (?) is a massive change so we really need to describe
> this
> > in a way that can be widely circulated.
> >
> > I actually think this would be a good idea. But there are a ton of
> > advantages to good old fashioned text files in terms of config management
> > and change control. And trying to support both may or may not be better.
> >
> > -Jay
> >
> >
> > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >
> > > Created a KIP
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > >
> > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > >
> > > /*******************************************
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Joe Stein <jo...@stealth.ly>.
There is still some to-dos to be done in
https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef
https://reviews.apache.org/r/30126/diff/ once that is in.
We can get more written up on it, will do.
On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <ja...@gmail.com> wrote:
> Hey Joe,
>
> Can you fill in this KIP? The purpose of these KIPs is to give a full
> overview of the feature, how it will work, be implemented, the
> considerations involved, etc. There is only like one sentence on this which
> isn't enough for anyone to know what you are thinking.
>
> Moving off of configs to something that I'm guessing would be
> Zookeeper-based (?) is a massive change so we really need to describe this
> in a way that can be widely circulated.
>
> I actually think this would be a good idea. But there are a ton of
> advantages to good old fashioned text files in terms of config management
> and change control. And trying to support both may or may not be better.
>
> -Jay
>
>
> On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Created a KIP
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> >
> > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Jay Kreps <ja...@gmail.com>.
Hey Joe,
Can you fill in this KIP? The purpose of these KIPs is to give a full
overview of the feature, how it will work, be implemented, the
considerations involved, etc. There is only like one sentence on this which
isn't enough for anyone to know what you are thinking.
Moving off of configs to something that I'm guessing would be
Zookeeper-based (?) is a massive change so we really need to describe this
in a way that can be widely circulated.
I actually think this would be a good idea. But there are a ton of
advantages to good old fashioned text files in terms of config management
and change control. And trying to support both may or may not be better.
-Jay
On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly> wrote:
> Created a KIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>
> JIRA https://issues.apache.org/jira/browse/KAFKA-1786
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Jay,
Could you elaborate a bit more regarding 2) above, what is the motivation
behind this?
On Wed, Mar 18, 2015 at 2:46 PM, Jay Kreps <ja...@gmail.com> wrote:
> The other question to figure out is the hierarchy of overrides. Currently
> some configs can be overridden at the topic level using a fairly ad hoc
> mechanism. I think Aditya's quota proposal requires having defaults and
> overrides at the client.id level. You could imagine similar requirements
> at
> the user level when we have user as a concept. You could also imagine
> overrides at a broker level (currently you do this by just changing the
> config on that broker only).
>
> With respect to dynamic update: Some configs (e.g. socket buffers) CAN'T be
> dynamically updated, and others (e.g. cleaner buffer) could theoretically
> be updated by the implementation would be quite complicated.
>
> I think do really get this right:
> 1. We would need to be able to mark configs as updatable or not (shouldn't
> be too hard after the conversion to the config def stuff) in a way that
> this automatically flows to the documentation. Obviously if you had to
> guess which config was updatable that would be a nightmare.
> 2. We will want to go with a pattern where instead of injecting individual
> values into the objects that need them from the config we pass some kind of
> Config object which has a reference to the current immutable config
> instance. To get a config you always have to dereference
> config.current.myConfigValue
> so that when current gets updated you get all the new configs atomically.
> 3. We will need to support a flexible notion of override that generalizes
> the LogConfig stuff we did at the topic level.
>
> This is all a fair amount of work to do well.
>
> -Jay
>
> On Wed, Mar 18, 2015 at 10:11 AM, Andrii Biletskyi <
> andrii.biletskyi@stealth.ly> wrote:
>
> > Aditya,
> >
> > Yes, you are right. We agreed to think how global config can
> > be done w/o restarting the broker, consumers/producers. I haven't
> > yet updated the KIP accordingly though.
> > In short, the problem is that KafkaConfig is "spread out" over different
> > components, like SocketServer, OffsetManager etc. This makes
> > it hard to dynamically change config, at least if we follow dynamic
> > LogConfig approach. So my opinion and vision is that we need to
> > treat each config block (like quotas in your case) separately. Logically,
> > different cases will be handled differently, some of the configs are
> > not eligible for dynamic change at all.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Wed, Mar 18, 2015 at 6:54 PM, Aditya Auradkar <
> > aauradkar@linkedin.com.invalid> wrote:
> >
> > > Hi Andrii,
> > >
> > > Thanks for the writeup.
> > > IMO, we should be able to support dynamically changing certain configs.
> > > For example: the Quota proposal relies on such a mechanism for changing
> > > quotas on the fly.
> > >
> > > Aditya
> > > ________________________________________
> > > From: Andrii Biletskyi [andrii.biletskyi@stealth.ly]
> > > Sent: Tuesday, March 03, 2015 10:30 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management
> > >
> > > Hey Jun,
> > >
> > > Initially I was thinking about instead of 3-4 state that global config
> > > changes
> > > take effect only after broker restart. So it's just:
> > >
> > > 3-4. On each broker startup apply global config from ZK
> > >
> > > In other words, the comprehensive workflow is the following:
> > > 1. Issue ChangeGlobalConfigRequest
> > > 2. Controller validates / stores config in ZK
> > > (out of scope of this KIP) 3. Do rolling restart for brokers one by one
> > to
> > > pick up
> > > config changes
> > >
> > > My understanding is that we won't be able to handle config change
> > > dynamically
> > > as we do for Log config. The main reason, from my point of view, is
> that
> > > broker config operates such settings as num.io.threads updating which
> > would
> > > mean
> > > gracefully restart some of the broker's components (in this case
> > > SocketServer) which is,
> > > in turn, might be very tricky.
> > >
> > > Thanks,
> > > Andrii Biletskyi
> > >
> > > On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > It seems the proposed workflow is the following.
> > > >
> > > > 1. Client issues a global config update request to the broker.
> > > > 2. Broker writes the new config to ZK.
> > > > 3. Controller picks up the changes from ZK.
> > > > 4. Controller propagates the config changes to all brokers.
> > > >
> > > > Do we need to add a new request/response to propagate the config
> > changes?
> > > >
> > > > Also, this logic is a bit different from how per topic config changes
> > > > works: each broker reads from ZK directly. It would be good to make
> > them
> > > > consistent.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > > wrote:
> > > >
> > > > > Created a KIP
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop
> >
> > > > > ********************************************/
> > > > >
> > > >
> > >
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Jay Kreps <ja...@gmail.com>.
The other question to figure out is the hierarchy of overrides. Currently
some configs can be overridden at the topic level using a fairly ad hoc
mechanism. I think Aditya's quota proposal requires having defaults and
overrides at the client.id level. You could imagine similar requirements at
the user level when we have user as a concept. You could also imagine
overrides at a broker level (currently you do this by just changing the
config on that broker only).
With respect to dynamic update: Some configs (e.g. socket buffers) CAN'T be
dynamically updated, and others (e.g. cleaner buffer) could theoretically
be updated by the implementation would be quite complicated.
I think do really get this right:
1. We would need to be able to mark configs as updatable or not (shouldn't
be too hard after the conversion to the config def stuff) in a way that
this automatically flows to the documentation. Obviously if you had to
guess which config was updatable that would be a nightmare.
2. We will want to go with a pattern where instead of injecting individual
values into the objects that need them from the config we pass some kind of
Config object which has a reference to the current immutable config
instance. To get a config you always have to dereference
config.current.myConfigValue
so that when current gets updated you get all the new configs atomically.
3. We will need to support a flexible notion of override that generalizes
the LogConfig stuff we did at the topic level.
This is all a fair amount of work to do well.
-Jay
On Wed, Mar 18, 2015 at 10:11 AM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Aditya,
>
> Yes, you are right. We agreed to think how global config can
> be done w/o restarting the broker, consumers/producers. I haven't
> yet updated the KIP accordingly though.
> In short, the problem is that KafkaConfig is "spread out" over different
> components, like SocketServer, OffsetManager etc. This makes
> it hard to dynamically change config, at least if we follow dynamic
> LogConfig approach. So my opinion and vision is that we need to
> treat each config block (like quotas in your case) separately. Logically,
> different cases will be handled differently, some of the configs are
> not eligible for dynamic change at all.
>
> Thanks,
> Andrii Biletskyi
>
> On Wed, Mar 18, 2015 at 6:54 PM, Aditya Auradkar <
> aauradkar@linkedin.com.invalid> wrote:
>
> > Hi Andrii,
> >
> > Thanks for the writeup.
> > IMO, we should be able to support dynamically changing certain configs.
> > For example: the Quota proposal relies on such a mechanism for changing
> > quotas on the fly.
> >
> > Aditya
> > ________________________________________
> > From: Andrii Biletskyi [andrii.biletskyi@stealth.ly]
> > Sent: Tuesday, March 03, 2015 10:30 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management
> >
> > Hey Jun,
> >
> > Initially I was thinking about instead of 3-4 state that global config
> > changes
> > take effect only after broker restart. So it's just:
> >
> > 3-4. On each broker startup apply global config from ZK
> >
> > In other words, the comprehensive workflow is the following:
> > 1. Issue ChangeGlobalConfigRequest
> > 2. Controller validates / stores config in ZK
> > (out of scope of this KIP) 3. Do rolling restart for brokers one by one
> to
> > pick up
> > config changes
> >
> > My understanding is that we won't be able to handle config change
> > dynamically
> > as we do for Log config. The main reason, from my point of view, is that
> > broker config operates such settings as num.io.threads updating which
> would
> > mean
> > gracefully restart some of the broker's components (in this case
> > SocketServer) which is,
> > in turn, might be very tricky.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > It seems the proposed workflow is the following.
> > >
> > > 1. Client issues a global config update request to the broker.
> > > 2. Broker writes the new config to ZK.
> > > 3. Controller picks up the changes from ZK.
> > > 4. Controller propagates the config changes to all brokers.
> > >
> > > Do we need to add a new request/response to propagate the config
> changes?
> > >
> > > Also, this logic is a bit different from how per topic config changes
> > > works: each broker reads from ZK directly. It would be good to make
> them
> > > consistent.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> > wrote:
> > >
> > > > Created a KIP
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > > >
> > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Andrii Biletskyi <an...@stealth.ly>.
Aditya,
Yes, you are right. We agreed to think how global config can
be done w/o restarting the broker, consumers/producers. I haven't
yet updated the KIP accordingly though.
In short, the problem is that KafkaConfig is "spread out" over different
components, like SocketServer, OffsetManager etc. This makes
it hard to dynamically change config, at least if we follow dynamic
LogConfig approach. So my opinion and vision is that we need to
treat each config block (like quotas in your case) separately. Logically,
different cases will be handled differently, some of the configs are
not eligible for dynamic change at all.
Thanks,
Andrii Biletskyi
On Wed, Mar 18, 2015 at 6:54 PM, Aditya Auradkar <
aauradkar@linkedin.com.invalid> wrote:
> Hi Andrii,
>
> Thanks for the writeup.
> IMO, we should be able to support dynamically changing certain configs.
> For example: the Quota proposal relies on such a mechanism for changing
> quotas on the fly.
>
> Aditya
> ________________________________________
> From: Andrii Biletskyi [andrii.biletskyi@stealth.ly]
> Sent: Tuesday, March 03, 2015 10:30 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management
>
> Hey Jun,
>
> Initially I was thinking about instead of 3-4 state that global config
> changes
> take effect only after broker restart. So it's just:
>
> 3-4. On each broker startup apply global config from ZK
>
> In other words, the comprehensive workflow is the following:
> 1. Issue ChangeGlobalConfigRequest
> 2. Controller validates / stores config in ZK
> (out of scope of this KIP) 3. Do rolling restart for brokers one by one to
> pick up
> config changes
>
> My understanding is that we won't be able to handle config change
> dynamically
> as we do for Log config. The main reason, from my point of view, is that
> broker config operates such settings as num.io.threads updating which would
> mean
> gracefully restart some of the broker's components (in this case
> SocketServer) which is,
> in turn, might be very tricky.
>
> Thanks,
> Andrii Biletskyi
>
> On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > It seems the proposed workflow is the following.
> >
> > 1. Client issues a global config update request to the broker.
> > 2. Broker writes the new config to ZK.
> > 3. Controller picks up the changes from ZK.
> > 4. Controller propagates the config changes to all brokers.
> >
> > Do we need to add a new request/response to propagate the config changes?
> >
> > Also, this logic is a bit different from how per topic config changes
> > works: each broker reads from ZK directly. It would be good to make them
> > consistent.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >
> > > Created a KIP
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> > >
> > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> > >
> > > /*******************************************
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> >
>
RE: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Hi Andrii,
Thanks for the writeup.
IMO, we should be able to support dynamically changing certain configs. For example: the Quota proposal relies on such a mechanism for changing quotas on the fly.
Aditya
________________________________________
From: Andrii Biletskyi [andrii.biletskyi@stealth.ly]
Sent: Tuesday, March 03, 2015 10:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-5 - Broker Configuration Management
Hey Jun,
Initially I was thinking about instead of 3-4 state that global config
changes
take effect only after broker restart. So it's just:
3-4. On each broker startup apply global config from ZK
In other words, the comprehensive workflow is the following:
1. Issue ChangeGlobalConfigRequest
2. Controller validates / stores config in ZK
(out of scope of this KIP) 3. Do rolling restart for brokers one by one to
pick up
config changes
My understanding is that we won't be able to handle config change
dynamically
as we do for Log config. The main reason, from my point of view, is that
broker config operates such settings as num.io.threads updating which would
mean
gracefully restart some of the broker's components (in this case
SocketServer) which is,
in turn, might be very tricky.
Thanks,
Andrii Biletskyi
On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <ju...@confluent.io> wrote:
> It seems the proposed workflow is the following.
>
> 1. Client issues a global config update request to the broker.
> 2. Broker writes the new config to ZK.
> 3. Controller picks up the changes from ZK.
> 4. Controller propagates the config changes to all brokers.
>
> Do we need to add a new request/response to propagate the config changes?
>
> Also, this logic is a bit different from how per topic config changes
> works: each broker reads from ZK directly. It would be good to make them
> consistent.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Created a KIP
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> >
> > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Andrii Biletskyi <an...@stealth.ly>.
Hey Jun,
Initially I was thinking about instead of 3-4 state that global config
changes
take effect only after broker restart. So it's just:
3-4. On each broker startup apply global config from ZK
In other words, the comprehensive workflow is the following:
1. Issue ChangeGlobalConfigRequest
2. Controller validates / stores config in ZK
(out of scope of this KIP) 3. Do rolling restart for brokers one by one to
pick up
config changes
My understanding is that we won't be able to handle config change
dynamically
as we do for Log config. The main reason, from my point of view, is that
broker config operates such settings as num.io.threads updating which would
mean
gracefully restart some of the broker's components (in this case
SocketServer) which is,
in turn, might be very tricky.
Thanks,
Andrii Biletskyi
On Tue, Mar 3, 2015 at 7:43 PM, Jun Rao <ju...@confluent.io> wrote:
> It seems the proposed workflow is the following.
>
> 1. Client issues a global config update request to the broker.
> 2. Broker writes the new config to ZK.
> 3. Controller picks up the changes from ZK.
> 4. Controller propagates the config changes to all brokers.
>
> Do we need to add a new request/response to propagate the config changes?
>
> Also, this logic is a bit different from how per topic config changes
> works: each broker reads from ZK directly. It would be good to make them
> consistent.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Created a KIP
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> >
> > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Jun Rao <ju...@confluent.io>.
It seems the proposed workflow is the following.
1. Client issues a global config update request to the broker.
2. Broker writes the new config to ZK.
3. Controller picks up the changes from ZK.
4. Controller propagates the config changes to all brokers.
Do we need to add a new request/response to propagate the config changes?
Also, this logic is a bit different from how per topic config changes
works: each broker reads from ZK directly. It would be good to make them
consistent.
Thanks,
Jun
On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <jo...@stealth.ly> wrote:
> Created a KIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>
> JIRA https://issues.apache.org/jira/browse/KAFKA-1786
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Gwen Shapira <gs...@cloudera.com>.
Thanks for clarifying the upgrade path.
+1 for this proposal, it will greatly reduce errors resulting from
configuration mismatches.
On Thu, Jan 22, 2015 at 8:40 AM, Joe Stein <jo...@stealth.ly> wrote:
> Deployment proposal:
>
> 1. If global configuration isn't being used, do what you do today (there
> should be configuration brokers.admin.global.configuration=true
> [default=false]. This works for the upgrade. Once all brokers are upgraded
> proceed.
> 2. if the global config is used then only some (host, port) will be able
> to be overridden in property for a single broker. There will always be
> configs that make sense for a single broker to be able to override. In the
> most case so far going through them there were few though. We should have
> this clear in the code for how to update those fields. This latter point is
> tied into https://issues.apache.org/jira/browse/KAFKA-1845 closely.
> 3. If the property is not set in storage mechanism then use default from
> how we manage the configuration definitions.
>
>
> On Thu, Jan 22, 2015 at 11:16 AM, Harsha <ka...@harsha.io> wrote:
>
>> Hi Joe,
>> How does initial deployments will look in this environment. I guess
>> users will deploy the kafka cluster and it will be running with
>> defaults and they can use a command line tool to update the
>> configs?.
>> I think you need to update the JIRA and mailing list thread links on the
>> KIP page.
>> Thanks,
>> Harsha
>>
>>
>>
>> On Wed, Jan 21, 2015, at 10:34 PM, Joe Stein wrote:
>> > Created a KIP
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>> >
>> > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
>> >
>> > /*******************************************
>> > Joe Stein
>> > Founder, Principal Consultant
>> > Big Data Open Source Security LLC
>> > http://www.stealth.ly
>> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> > ********************************************/
>>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Joe Stein <jo...@stealth.ly>.
Deployment proposal:
1. If global configuration isn't being used, do what you do today (there
should be configuration brokers.admin.global.configuration=true
[default=false]. This works for the upgrade. Once all brokers are upgraded
proceed.
2. if the global config is used then only some (host, port) will be able
to be overridden in property for a single broker. There will always be
configs that make sense for a single broker to be able to override. In the
most case so far going through them there were few though. We should have
this clear in the code for how to update those fields. This latter point is
tied into https://issues.apache.org/jira/browse/KAFKA-1845 closely.
3. If the property is not set in storage mechanism then use default from
how we manage the configuration definitions.
On Thu, Jan 22, 2015 at 11:16 AM, Harsha <ka...@harsha.io> wrote:
> Hi Joe,
> How does initial deployments will look in this environment. I guess
> users will deploy the kafka cluster and it will be running with
> defaults and they can use a command line tool to update the
> configs?.
> I think you need to update the JIRA and mailing list thread links on the
> KIP page.
> Thanks,
> Harsha
>
>
>
> On Wed, Jan 21, 2015, at 10:34 PM, Joe Stein wrote:
> > Created a KIP
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
> >
> > JIRA https://issues.apache.org/jira/browse/KAFKA-1786
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
>
Re: [DISCUSS] KIP-5 - Broker Configuration Management
Posted by Harsha <ka...@harsha.io>.
Hi Joe,
How does initial deployments will look in this environment. I guess
users will deploy the kafka cluster and it will be running with
defaults and they can use a command line tool to update the
configs?.
I think you need to update the JIRA and mailing list thread links on the
KIP page.
Thanks,
Harsha
On Wed, Jan 21, 2015, at 10:34 PM, Joe Stein wrote:
> Created a KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management
>
> JIRA https://issues.apache.org/jira/browse/KAFKA-1786
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/