[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id

["Joe Stein (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085808#comment-14085808 ] 

Joe Stein edited comment on KAFKA-1070 at 8/5/14 5:06 AM:
----------------------------------------------------------

This sounds like a very cool and VERY useful feature. Excited to use it myself often.

I know of a few (>10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing "." in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." is replaced). 

I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup.  It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace "." is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it?  I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that.  We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate their scripts but not sure they would do it and if they did would have to decommission brokers which in a prod environment could take a few weeks/months.  If we let them start at 1 and exclude what they have then they can do it one at a time.  After taking down the first broker and bring it back up (empty) it is broker.id=1, and so on (and if they have a 5 they don't have to take it down), etc.






was (Author: joestein):
This sounds like a very cool and VERY useful feature. Excited to use it myself often.

I know of a few (>10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing "." in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." is replaced). 

I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup.  It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace "." is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it?  I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that.  We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate off what they have and then moving forward in their automation deal with the lower number or something.  It is hard to say how folks find creative solutions to common problems without every speaking to each other.  I don't know how this will work for them though.



> Auto-assign node id
> -------------------
>
>                 Key: KAFKA-1070
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1070
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jay Kreps
>            Assignee: Sriharsha Chintalapani
>              Labels: usability
>         Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch, KAFKA-1070_2014-07-22_11:34:18.patch, KAFKA-1070_2014-07-24_20:58:17.patch, KAFKA-1070_2014-07-24_21:05:33.patch
>
>
> It would be nice to have Kafka brokers auto-assign node ids rather than having that be a configuration. Having a configuration is irritating because (1) you have to generate a custom config for each broker and (2) even though it is in configuration, changing the node id can cause all kinds of bad things to happen.



--
This message was sent by Atlassian JIRA
(v6.2#6252)