You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jon Bender <jo...@gmail.com> on 2014/05/14 20:45:19 UTC
Leader for almost all topic, partitions on the same broker
Hello,
I have a 3-node cluster that has had a couple issues lately. One thing I'm
trying to sort out is why the topic-partitions are all owned by a single
leader (when i list topics, the leader is assigned to the current
controller node, irrespective the preferred replica).
I have tried to use the preferred replica election tool per:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
But I don't see any change taking place.
Only thing I see in the logs on this machine is:
[2014-05-14 11:37:22,146] 69818066
[ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
INFO kafka.utils.ZkUtils$ - conflict in /controller data: {
"brokerid":1390348134, "timestamp":"1400026164986", "version":1 } stored
data: { "brokerid":1390348134, "timestamp":"1400026164512", "version":1 }
[2014-05-14 11:37:22,147] 69818067
[ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
INFO kafka.utils.ZkUtils$ - I wrote this conflicted ephemeral node [{
"brokerid":1390348134, "timestamp":"1400026164986", "version":1 }] at
/controller a while back in a different session, hence I will backoff for
this node to be deleted by Zookeeper and retry
Kafka version is 0.8.0.
Any suggestions on how to get this cluster to properly rebalance?
Cheers,
Jon
Re: Leader for almost all topic, partitions on the same broker
Posted by Jun Rao <ju...@gmail.com>.
Does that error occur continuously? Preferred leaders are the first replica
in the assigned replica list. Could you list the topics and see the
distribution of the first replica in all partitions?
Thanks,
Jun
On Wed, May 14, 2014 at 11:45 AM, Jon Bender <jo...@gmail.com>wrote:
> Hello,
>
> I have a 3-node cluster that has had a couple issues lately. One thing I'm
> trying to sort out is why the topic-partitions are all owned by a single
> leader (when i list topics, the leader is assigned to the current
> controller node, irrespective the preferred replica).
>
> I have tried to use the preferred replica election tool per:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
>
> But I don't see any change taking place.
>
> Only thing I see in the logs on this machine is:
>
> [2014-05-14 11:37:22,146] 69818066
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - conflict in /controller data: {
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 } stored
> data: { "brokerid":1390348134, "timestamp":"1400026164512", "version":1 }
> [2014-05-14 11:37:22,147] 69818067
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - I wrote this conflicted ephemeral node [{
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 }] at
> /controller a while back in a different session, hence I will backoff for
> this node to be deleted by Zookeeper and retry
>
> Kafka version is 0.8.0.
>
> Any suggestions on how to get this cluster to properly rebalance?
>
> Cheers,
> Jon
>
Re: Leader for almost all topic, partitions on the same broker
Posted by Jon Bender <jo...@gmail.com>.
Thanks for the responses!
>Does that error occur continuously? Preferred leaders are the first replica
>in the assigned replica list. Could you list the topics and see the
>distribution of the first replica in all partitions?
The state we were in showed all topics led by a single broker. The
preferred replica (first in the isr list) was fairly evenly distributed
among brokers, but as mentioned we could not move the leaders from the
single broker to whichever was preferred for that topic-partition even with
the replica election tool.
>Did you see all 3 brokers registered in ZK? From the error logs it seems
>some of the brokers did not successfully startup and hence cannot take any
>partitions.
We had actually been running the system for a while without problems.
One of the main culprits seems to be long GC pauses leading to Zookeeper
timeouts on different nodes. After this all leaders snap to a single
broker. At some point this became unrecoverable, and everything was stuck
on a single broker and couldn't be moved. We did a full restart of all
brokers at once (doing so in a rolling fashion didn't seem to make a
difference) and eventually we were able to at least get the cluster to get
into a state where we could reassign leaders using that tool. However,
after another big GC pause we would run into the same issue of all topics
gravitating to a single broker.
We're doing a couple things to try and avoid this scenario: move to Java 7
/ G1 garbage collection to see if we can avoid these costly pauses, and
ultimately upgrade to 0.8.1.1.
Regards,
Jon
On Fri, May 16, 2014 at 9:32 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Jon,
>
> Did you see all 3 brokers registered in ZK? From the error logs it seems
> some of the brokers did not successfully startup and hence cannot take any
> partitions.
>
> Guozhang
>
>
> On Wed, May 14, 2014 at 11:45 AM, Jon Bender <jonathan.bender@gmail.com
> >wrote:
>
> > Hello,
> >
> > I have a 3-node cluster that has had a couple issues lately. One thing
> I'm
> > trying to sort out is why the topic-partitions are all owned by a single
> > leader (when i list topics, the leader is assigned to the current
> > controller node, irrespective the preferred replica).
> >
> > I have tried to use the preferred replica election tool per:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
> >
> > But I don't see any change taking place.
> >
> > Only thing I see in the logs on this machine is:
> >
> > [2014-05-14 11:37:22,146] 69818066
> > [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> > INFO kafka.utils.ZkUtils$ - conflict in /controller data: {
> > "brokerid":1390348134, "timestamp":"1400026164986", "version":1 } stored
> > data: { "brokerid":1390348134, "timestamp":"1400026164512", "version":1 }
> > [2014-05-14 11:37:22,147] 69818067
> > [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> > INFO kafka.utils.ZkUtils$ - I wrote this conflicted ephemeral node [{
> > "brokerid":1390348134, "timestamp":"1400026164986", "version":1 }] at
> > /controller a while back in a different session, hence I will backoff for
> > this node to be deleted by Zookeeper and retry
> >
> > Kafka version is 0.8.0.
> >
> > Any suggestions on how to get this cluster to properly rebalance?
> >
> > Cheers,
> > Jon
> >
>
>
>
> --
> -- Guozhang
>
Re: Leader for almost all topic, partitions on the same broker
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Jon,
Did you see all 3 brokers registered in ZK? From the error logs it seems
some of the brokers did not successfully startup and hence cannot take any
partitions.
Guozhang
On Wed, May 14, 2014 at 11:45 AM, Jon Bender <jo...@gmail.com>wrote:
> Hello,
>
> I have a 3-node cluster that has had a couple issues lately. One thing I'm
> trying to sort out is why the topic-partitions are all owned by a single
> leader (when i list topics, the leader is assigned to the current
> controller node, irrespective the preferred replica).
>
> I have tried to use the preferred replica election tool per:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
>
> But I don't see any change taking place.
>
> Only thing I see in the logs on this machine is:
>
> [2014-05-14 11:37:22,146] 69818066
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - conflict in /controller data: {
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 } stored
> data: { "brokerid":1390348134, "timestamp":"1400026164512", "version":1 }
> [2014-05-14 11:37:22,147] 69818067
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - I wrote this conflicted ephemeral node [{
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 }] at
> /controller a while back in a different session, hence I will backoff for
> this node to be deleted by Zookeeper and retry
>
> Kafka version is 0.8.0.
>
> Any suggestions on how to get this cluster to properly rebalance?
>
> Cheers,
> Jon
>
--
-- Guozhang
Re: Leader for almost all topic, partitions on the same broker
Posted by Neha Narkhede <ne...@gmail.com>.
Try using the kafka-topics.sh tool with the --describe option to see if the
1st replica is well distributed across all topics and partitions. The
preferred replica election tool just moves the leader to the 1st replica in
the replica list for a partition.
Also, we fixed a few bugs related to this tool. It might be worth trying
the latest stable release (0.8.1.1)
Thanks,
Neha
On Wed, May 14, 2014 at 11:45 AM, Jon Bender <jo...@gmail.com>wrote:
> Hello,
>
> I have a 3-node cluster that has had a couple issues lately. One thing I'm
> trying to sort out is why the topic-partitions are all owned by a single
> leader (when i list topics, the leader is assigned to the current
> controller node, irrespective the preferred replica).
>
> I have tried to use the preferred replica election tool per:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
>
> But I don't see any change taking place.
>
> Only thing I see in the logs on this machine is:
>
> [2014-05-14 11:37:22,146] 69818066
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - conflict in /controller data: {
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 } stored
> data: { "brokerid":1390348134, "timestamp":"1400026164512", "version":1 }
> [2014-05-14 11:37:22,147] 69818067
> [ZkClient-EventThread-21-my.server1:2181,my.server2:2181,my.server3:2181]
> INFO kafka.utils.ZkUtils$ - I wrote this conflicted ephemeral node [{
> "brokerid":1390348134, "timestamp":"1400026164986", "version":1 }] at
> /controller a while back in a different session, hence I will backoff for
> this node to be deleted by Zookeeper and retry
>
> Kafka version is 0.8.0.
>
> Any suggestions on how to get this cluster to properly rebalance?
>
> Cheers,
> Jon
>