You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Arup Malakar (JIRA)" <ji...@apache.org> on 2016/10/25 21:59:59 UTC
[jira] [Commented] (KAFKA-1466) Kafka server is hung after throwing
"Attempt to swap the new high watermark file with the old one failed"
[ https://issues.apache.org/jira/browse/KAFKA-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606600#comment-15606600 ]
Arup Malakar commented on KAFKA-1466:
-------------------------------------
I think we may have found the cause for this issue on our systems (after two and half years). The chef recipe used to deploy the server had a bug and was deleting the configuration directory and recreating it. Luckily with Kafka's replication etc it wasn't a problem and we didn't see issues except occasional exception in our logs.
> Kafka server is hung after throwing "Attempt to swap the new high watermark file with the old one failed"
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-1466
> URL: https://issues.apache.org/jira/browse/KAFKA-1466
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Arup Malakar
> Attachments: kafka.log.1
>
>
> We have a kafka cluster of four nodes. The cluster was down after one of the nodes threw the following error:
> 2014-05-21 23:19:44 FATAL [highwatermark-checkpoint-thread1]: HighwaterMarkCheckpoint:109 - Attempt to swap the new high watermark file with the old one failed. I saw the following message in the log file of the failed node:
> {code}
> 2014-05-21 23:19:44 FATAL [highwatermark-checkpoint-thread1]: HighwaterMarkCheckpoint:109 - Attempt to swap the new high watermark file with the old one failed
> 2014-05-21 23:19:44 INFO [Thread-1]: KafkaServer:67 - [Kafka Server 4], Shutting down
> 2014-05-21 23:19:44 INFO [Thread-1]: KafkaZooKeeper:67 - Closing zookeeper client...
> 2014-05-21 23:19:44 INFO [ZkClient-EventThread-21-zoo-c2n1.us-east-1.ooyala.com,zoo-c2n2.us-east-1.ooyala.com,zoo-c2n3.us-east-1.ooyala.com,zoo-c2n4.us-east-1.ooyala.com,zoo-c2n5.u
> s-east-1.ooyala.com]: ZkEventThread:82 - Terminate ZkClient event thread.
> 2014-05-21 23:19:44 INFO [main-EventThread]: ClientCnxn:521 - EventThread shut down
> 2014-05-21 23:19:44 INFO [Thread-1]: ZooKeeper:544 - Session: 0x1456b562865b172 closed
> 2014-05-21 23:19:44 INFO [kafka-processor-9092-0]: Processor:67 - Closing socket connection to /10.245.173.136.
> 2014-05-21 23:19:44 INFO [Thread-1]: SocketServer:67 - [Socket Server on Broker 4], Shutting down
> 2014-05-21 23:19:44 INFO [Thread-1]: SocketServer:67 - [Socket Server on Broker 4], Shutdown completed
> 2014-05-21 23:19:44 INFO [Thread-1]: KafkaRequestHandlerPool:67 - [Kafka Request Handler on Broker 4], shutting down
> 2014-05-21 23:19:44 INFO [Thread-1]: KafkaRequestHandlerPool:67 - [Kafka Request Handler on Broker 4], shutted down completely
> 2014-05-21 23:19:44 INFO [Thread-1]: KafkaScheduler:67 - Shutdown Kafka scheduler
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaManager:67 - [Replica Manager on Broker 4]: Shut down
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherManager:67 - [ReplicaFetcherManager on broker 4] shutting down
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-3], Shutting down
> 2014-05-21 23:19:45 INFO [ReplicaFetcherThread-0-3]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-3], Stopped
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-3], Shutdown completed
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-2], Shutting down
> 2014-05-21 23:19:45 INFO [ReplicaFetcherThread-0-2]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-2], Stopped
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherThread:67 - [ReplicaFetcherThread-0-2], Shutdown completed
> 2014-05-21 23:19:45 INFO [Thread-1]: ReplicaFetcherManager:67 - [ReplicaFetcherManager on broker 4] shutdown completed
> {code}
> I notice that since this error was logged there weren't any more logs in the log file but the process was still alive, so I guess it was hung.
> The other nodes in the cluster was not able to recover from this error. The partitions owned by this failed node had its leader set to -1:
> {code}
> topic: test_topic partition: 8 leader: -1 replicas: 4 isr:
> {code}
> And other nodes were continuously logging the following errors in the log file:
> {code}
> 2014-05-22 20:03:28 ERROR [kafka-request-handler-7]: KafkaApis:102 - [KafkaApi-3] Error while fetching metadata for partition [test_topic,8]
> kafka.common.LeaderNotAvailableException: Leader not available for partition [test_topic,8]
> at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:474)
> at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:462)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:462)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:458)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:123)
> at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:322)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.HashSet.map(HashSet.scala:32)
> at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:458)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> I had to restart the failed kafka node to recover the cluster. We expect the kafka cluster to work even if a node is down. Any clue what went wrong here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)