[jira] [Created] (KAFKA-4418) Broker Leadership Election Fails If Missing ZK Path Raises Exception

["Michael Pedersen (JIRA)" <ji...@apache.org>]

Michael Pedersen created KAFKA-4418:
---------------------------------------

             Summary: Broker Leadership Election Fails If Missing ZK Path Raises Exception
                 Key: KAFKA-4418
                 URL: https://issues.apache.org/jira/browse/KAFKA-4418
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.10.0.1, 0.10.0.0, 0.9.0.1
            Reporter: Michael Pedersen


Our Kafka cluster went down because a single node went down *and* a path in Zookeeper was missing for one topic (/brokers/topics/<topicname>/partitions). When this occurred, leadership election could not run, and produced a stack trace that looked like this:

Failed to start preferred replica election
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/warandpeace/partitions
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:995)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:675)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:671)
	at kafka.utils.ZkUtils.getChildren(ZkUtils.scala:537)
	at kafka.utils.ZkUtils$$anonfun$getAllPartitions$1.apply(ZkUtils.scala:817)
	at kafka.utils.ZkUtils$$anonfun$getAllPartitions$1.apply(ZkUtils.scala:816)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at kafka.utils.ZkUtils.getAllPartitions(ZkUtils.scala:816)
	at kafka.admin.PreferredReplicaLeaderElectionCommand$.main(PreferredReplicaLeaderElectionCommand.scala:64)
	at kafka.admin.PreferredReplicaLeaderElectionCommand.main(PreferredReplicaLeaderElectionCommand.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/warandpeace/partitions
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
	at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:114)
	at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:678)
	at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:675)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:985)
	... 16 more

I have checked through the code a bit, and have found a quick place to introduce a fix that would seem to allow the leadership election to continue. Specifically, the function at https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L633 does not handle possible exceptions. Wrapping a try/catch block here would work, but could introduce a number of other problems:

* If the code is used elsewhere, the exception might be needed at a higher level to prevent something else.
* Unless the exception is logged/reported somehow, no one will know this problem exists, which makes debugging other problems harder.

I'm sure there are other issues I'm not aware of, but those two come to mind quickly. What would be the best route for getting this resolved quickly?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)