Re: dynamic config zookeeper code

[Jay Kreps <ja...@gmail.com>]

Hey Joe,

I threw up a first draft on this patch, take a look and see what you think.

I think what you are saying is that you want to avoid having configuration
changes take effect dynamically on all running servers. I think this is
pretty easy to accomplish, we can just make it possible to disable the
TopicConfigManager. If we disable this then running the command to alter a
topic config override would not have any immediate impact other than
changing the value in zookeeper and would only take effect on individual
servers as they were bounced (as today). Let me know if that would work for
you and I will add an topic.config.updates.enable={true,false}
configuration to allow this.

-Jay


On Wed, Jan 23, 2013 at 10:19 PM, Joe Stein <cr...@gmail.com> wrote:

> TopicConfigWatcher sounds good
>
> I still feel there needs to be some abstraction to TopicConfigWather
> because using Zookeeper Watcher means having global changes to a cluster
> and in production we don't allow global updates to every server so i am
> going to have to use another mechanism.  How we do many configuration
> changes in production is to notify a server that ZK has a change and that
> one server goes and gets the changes so we can do sequential configuration
> changes with some sanity check for each server change is alright. one way
> we do this is responding to http requests with name=value (i.e.
> http://serverXYZ/admin/config=reload then hit a health check on
> serverXYZ).
>  for scala servers we embed jetty.
>
> if you can abstract the watcher that would be awesome or I can abstract it
> after you are done and build another update.  I am not sold my way is
> better for anyone that the ZK watcher I actually like that approach but if
> others need or want the feature (like we would but i don't mind rolling a
> patch into production that is not in the main code base) then we can agree
> to support both methods.
>
> also what about having a new package
>
> kafka/config
>
> this new class goes there
>
> and another ticket refactoring other configs into there?  am I getting too
> off my rocker to suggest re-packing kafka to org.apache.kafka? I don't mind
> the code change in the kafka code but the downstream clients may freek...
> but if we want to-do it now would be the time to-do it me thinks
>
> On Thu, Jan 24, 2013 at 12:40 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Where should the zk code for the dynamic topic config go? I am not sure I
> > understand the overall lay of the land well enough to know if I am doing
> > the right thing.
> >
> > For instance, what the is KafkaZookeeper? What is its domain of
> > responsibility? Should all zk code go there? Lots of other zk code seems
> > elsewhere, KafkaZookeeper just seems to be the znode for broker liveness
> > (perhaps it should be renamed?). Why does some zk stuff go in
> > KafkaZookeeper and other stuff elsewhere? Actually in practice lots of it
> > seems to be stuck in ZkUtils as static methods, is there a reason for
> this?
> >
> > Maybe this stuff needs a maintainer? Or does it fall under "replication"?
> >
> > What I am thinking of doing is making a new class TopicConfigWatcher
> which
> > has a readAllLogConfigs() method and a watcher on the config change path.
> > This class would update a map of configs which would be shared with
> > LogManager to handle "alter topic". "create topic" can just do the read
> > from zk in response to the controller. Does this sound reasonable?
> >
> > -Jay
> >
>
>
>
> --
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>