[jira] [Updated] (AMBARI-12678) Installing additional Kafka broker on node with hostname that comes alphabetically before existing broker fails to start

["JuanFra Rodriguez (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMBARI-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

JuanFra Rodriguez updated AMBARI-12678:
---------------------------------------
    Affects Version/s: 1.7.0

> Installing additional Kafka broker on node with hostname that comes alphabetically before existing broker fails to start
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-12678
>                 URL: https://issues.apache.org/jira/browse/AMBARI-12678
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 1.7.0, 2.0.0, 2.1.0
>            Reporter: Carita Ou
>            Priority: Minor
>
> 1. Install Ambari with 3 nodes, example hostname01, hostname02, hostname03
> 2. Install Kafka service on hostname03. Kafka starts successfully
> 3. Go to hostname02 and add Kafka Broker. Start kafka broker and it will fail with below error:
> {code}
> [2015-08-05 17:13:51,306] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
> 		at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:174)
> 		at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:58)
> 		at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
> 		at kafka.server.KafkaServer.startup(KafkaServer.scala:124)
> 		at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
> 		at kafka.Kafka$.main(Kafka.scala:46)
> 		at kafka.Kafka.main(Kafka.scala)
> {code}
> The 2nd broker added on hostname02 failed to start because it was assigned the same brokerid as the broker installed on hostname03. The brokerid is assigned to a broker in kafka.py:
> {code}
> brokerid = str(sorted(params.kafka_hosts).index(params.hostname))
> kafka_server_config['broker.id'] = brokerid
> {code}
> params.py gets the kafka_hosts from clusterHostInfo:
> {code}
> kafka_hosts = config['clusterHostInfo']['kafka_broker_hosts']
> {code}
> clusterHostInfo maintains the list of kafka brokers in order:
> {code}
> "kafka_broker_hosts": [
> 	"hostname02",
> 	"hostname03"
> ],
> {code}
> Since the brokerid is the index of the hostname, and the broker on hostname03 was added first, it was at index 0. After adding a new broker on hostname02, it was inserted to the list at index 0 and hostname03 was moved to index 1. Because the broker on hostname03 was not restarted, it was still running with brokerid 0, so the broker on hostname02 with brokerid 0 was unable to start.
> Workaround: Restart all brokers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)