Re: It's 5.41am, we're after 20+ hours of debugging our prod cluster. See NotAssignedReplicaException and UnknownException errors. Help?

[Manish Sharma <ma...@gmail.com>]

Same here..
Started running into similar situations almost weekly ever since we
increased the number of partitions on some topics from 6 to 15 and added 3
brokers to our kafka cluster.

Last night I stopped all producers & consumers; restarted the brokers &
zookeepers and restarted producers/consumers.

Today morning I see an endless loop of shrinking ISR -> cached zkVersion
mismatch skip udating ISR again.

*[2015-11-07 11:55:47,260] INFO Partition [Wmt_Saturday_234,10] on broker
0: Shrinking ISR for partition [Wmt_Saturday_234,10] from 0,1 to 0
(kafka.cluster.Partition)*

*[2015-11-07 11:55:47,267] INFO Partition [Wmt_Saturday_234,10] on broker
0: Cached zkVersion [10] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)*




On Tue, Oct 20, 2015 at 9:45 AM, Shaun Senecal <sh...@lithium.com>
wrote:

> I can't say this is the same issue, but it sounds similar to a situation
> we experienced with Kafka 0.8.2.[1-2].  After restarting a broker, the
> cluster would never really recover (ISRs constantly changing, replication
> failing, etc).  We found the only way to fully recover the cluster was to
> stop all producers and consumers, restart the kafka cluster, the once the
> cluster was back up, restart the producers/consumers.  Obviously thats not
> acceptable for a production cluster, but that was the only thing we could
> find that would get us going again.
>
>
> Shaun
>
> ________________________________________
> From: Szymon Sobczak <sz...@getbase.com>
> Sent: October 19, 2015 9:52 PM
> To: users@kafka.apache.org
> Cc: Big Data
> Subject: It's 5.41am, we're after 20+ hours of debugging our prod cluster.
> See NotAssignedReplicaException and UnknownException errors. Help?
>
> Hi!
>
> We're running a 5-machine production Kafka cluster on version 0.8.1.1.
> Yesterday we had some disk problems on one of the replicas and decided to
> replace that node with a clean one. That's when we started experiencing
> many different problems:
>
> - partition replicas are still assigned to the old node and we can't remove
> it form the replica list
> - replicas are lagging behind, most of the topics have only one ISR
> - most of the leaders are on a single node
> - CPU load on the machines is constantly high
>
> We've tried to rebalance the cluster by moving the leaders, decreasing
> number of replicas and some others, but it doesn't seem to help. In the
> meantime I've noticed very weird errors in the kafka.log
>
> For partition 0 of topic product_templates with the following description:
>
> Topic:product_templates PartitionCount:2 ReplicationFactor:3 Configs:
> Topic: product_templates Partition: 0 Leader: 135 Replicas: 135,163,68 Isr:
> 135,68,163
> Topic: product_templates Partition: 1 Leader: 155 Replicas: 163,68,164 Isr:
> 155,68,164
>
> On machine 135 (which is a leader of product_templates,0) in kafka.log I
> see:
>
> kafka.common.NotAssignedReplicaException: Leader 135 failed to record
> follower 155's position 0 for partition [product_templates,0] since the
> replica 155 is not recognized to be one of the assigned replicas 68,163,135
> for partition [product_templates,0]
>
> And the complimentary, on 155 - NOT a replica product_templates,0:
>
> ERROR [ReplicaFetcherThread-0-135] 2015-10-20 04:41:47,011 Logging.scala
> kafka.server.ReplicaFetcherThread [ReplicaFetcherThread-0-135], Error for
> partition [product_templates,0] to broker 135:class
> kafka.common.UnknownException
>
> Both of those happen for multiple topics, on multiple machines. Every
> single one happens multiple times per second...
>
> How to approach this? Any help is appreciated!
>
> Thanks!
> Szymon.
>