You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rajasekar Elango <re...@salesforce.com> on 2013/09/13 01:56:45 UTC
Leader doesn't get assigned for new topics
We are seeing a problem that we we try to send messages to new topic it
fails kafka.common.LeaderNotAvailableException. But usually this problem
will be transient and if we re-send messages to same topic will work. But
now we tried rending message to same topic several time, but still fails
with same error:
In the server log I see ] Auto creation of topic test-sjl2 with 8
partitions and replication factor 3 is successful!. But listTopics command
shows leader "none" like below:
topic: test-sjl2 partition: 0 leader: none replicas: 2,4,3 isr:
topic: test-sjl2 partition: 1 leader: none replicas: 3,2,4 isr:
topic: test-sjl2 partition: 2 leader: none replicas: 4,3,2 isr:
topic: test-sjl2 partition: 3 leader: none replicas: 2,3,4 isr:
topic: test-sjl2 partition: 4 leader: none replicas: 3,4,2 isr:
topic: test-sjl2 partition: 5 leader: none replicas: 4,2,3 isr:
topic: test-sjl2 partition: 6 leader: none replicas: 2,4,3 isr:
topic: test-sjl2 partition: 7 leader: none replicas: 3,2,4 isr:
I also see following NotLeaderForPatritionExcetion and ZookeeperExcetion in
logs
kafka.common.NotLeaderForPartitionException
at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
at kafka.utils.Logging$class.warn(Logging.scala:88)
at kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
(kafka.utils.ZkUtils$) - Conditional update of path
/brokers/topics/FunnelProto/partitions/4/state with data {
"controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
"version":1 } and expected version 14 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/FunnelProto/partitions/4/state
2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
(kafka.utils.ZkUtils$) - Conditional update of path
/brokers/topics/FunnelProto/partitions/4/state with data {
"controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
"version":1 } and expected version 14 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/FunnelProto/partitions/4/state
Any clues on what could be problem.. ?
Any for your help.
--
Thanks,
Raja.
Re: Leader doesn't get assigned for new topics
Posted by Neha Narkhede <ne...@gmail.com>.
Do you see any errors in the controller.log or the state change log?
Thanks,
Neha
On Sep 12, 2013 10:47 PM, "Rajasekar Elango" <re...@salesforce.com> wrote:
> We are seeing a problem that we we try to send messages to new topic it
> fails kafka.common.LeaderNotAvailableException. But usually this problem
> will be transient and if we re-send messages to same topic will work. But
> now we tried rending message to same topic several time, but still fails
> with same error:
>
> In the server log I see ] Auto creation of topic test-sjl2 with 8
> partitions and replication factor 3 is successful!. But listTopics command
> shows leader "none" like below:
>
> topic: test-sjl2 partition: 0 leader: none replicas: 2,4,3
> isr:
> topic: test-sjl2 partition: 1 leader: none replicas: 3,2,4
> isr:
> topic: test-sjl2 partition: 2 leader: none replicas: 4,3,2
> isr:
> topic: test-sjl2 partition: 3 leader: none replicas: 2,3,4
> isr:
> topic: test-sjl2 partition: 4 leader: none replicas: 3,4,2
> isr:
> topic: test-sjl2 partition: 5 leader: none replicas: 4,2,3
> isr:
> topic: test-sjl2 partition: 6 leader: none replicas: 2,4,3
> isr:
> topic: test-sjl2 partition: 7 leader: none replicas: 3,2,4
> isr:
>
> I also see following NotLeaderForPatritionExcetion and ZookeeperExcetion in
> logs
>
> kafka.common.NotLeaderForPartitionException
> at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> Source)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> at kafka.utils.Logging$class.warn(Logging.scala:88)
> at kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> at
>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> (kafka.utils.ZkUtils$) - Conditional update of path
> /brokers/topics/FunnelProto/partitions/4/state with data {
> "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> "version":1 } and expected version 14 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> (kafka.utils.ZkUtils$) - Conditional update of path
> /brokers/topics/FunnelProto/partitions/4/state with data {
> "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> "version":1 } and expected version 14 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/FunnelProto/partitions/4/state
>
>
> Any clues on what could be problem.. ?
>
> Any for your help.
>
> --
> Thanks,
> Raja.
>
Re: Leader doesn't get assigned for new topics
Posted by Neha Narkhede <ne...@gmail.com>.
When new topics are created, it takes some time for the controller to
communicate the partition assignment for the new topic to all brokers.
Until that happens, attempts to send/receive data to the brokers fails with
the following kind of error -
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [jmx,3] failed due to
Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
But eventually the broker receives the updated metadata for the new topic
from the controller. When that happens, these errors go away.
This can also happen on a newly restarted broker if it was not shutdown
using controlled shutdown.
Thanks,
Neha
On Wed, Sep 18, 2013 at 10:30 AM, Rajasekar Elango
<re...@salesforce.com>wrote:
> From the output of StateChangeLogMerger tool, I see only this error
> repeated;
>
> [2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
> for partition [FunnelProto,0] (kafka.server.KafkaApis)
>
> On the state-change.log itself, I see this error:
>
> [2013-09-18 14:22:48,954] ERROR Conditional update of path
> /brokers/topics/test-1379439240191/partitions/2/state with data {
> "controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
> "version":1 } and expected version 8 fai
> led due to org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode = BadVersion for
> /brokers/topics/test-1379439240191/partitions/2/state
> (kafka.utils.ZkUtils$)
>
> Do you know reason for above error..? Also this problem seem to be
> intermittent, it started working now without any changes. I will continue
> to monitor.
>
> Thanks,
> Raja.
>
>
> On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Raja,
> >
> > Could you run the StateChangeLogMerger tool and give it one
> topic-partition
> > that has the above mentioned problem. This tool is documented here -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> > .
> >
> > Let me know if you run into any issues while using it.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> > <re...@salesforce.com>wrote:
> >
> > > Neha/Jun,
> > >
> > > The same problem started happening again although now our zookeeper
> > cluster
> > > is configured correctly. The produce always failed with
> > > LeaderNotAvailableException and list topics shows topic is created with
> > > leader "none". In the controller and stage-change log, I am seeing lot
> of
> > > these failures..
> > >
> > >
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client on partition [FunnelProto,6]
> failed
> > > due to Partition [FunnelProto,6] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client on partition [internal_metrics,3]
> > > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client on partition [FunnelProto,0]
> failed
> > > due to Partition [FunnelProto,0] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client on partition [jmx,3] failed due
> to
> > > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client on partition [FunnelProto,5]
> failed
> > > due to Partition [FunnelProto,5] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > >
> > >
> > > When I ran listTopics command for one of above topic, all partitions
> are
> > > under replicated (we have replication factor set to 3). Any clues on
> what
> > > could be issue and how can we get it back to working?
> > >
> > > Thanks,
> > > Raja.
> > >
> > >
> > >
> > > On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Ah ok. Thanks for sharing that.
> > > >
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> > > relango@salesforce.com
> > > > >wrote:
> > > >
> > > > > We have 3 zookeeper node in the cluster with a hardware load
> > balancer .
> > > > In
> > > > > one of the zookeeper, we did not configure ensemble correctly
> > (server.n
> > > > > property in zoo.cfg) . So it ended up as like 2 nodes in one
> cluster,
> > > one
> > > > > node in other cluster. The load balancer is randomly hitting one
> of 2
> > > > > zookeepers in two different cluster.
> > > > >
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> > > neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Just curious to know, what was the misconfiguration?
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > > > > > <re...@salesforce.com>wrote:
> > > > > >
> > > > > > > Thanks Neha and Jun, It turned out to be miss configuration in
> > our
> > > > > > > zookeeper cluster. After correcting it everything looks good.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Raja.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Any error in the controller and the state-change log? Are
> > brokers
> > > > > 2,3,4
> > > > > > > > alive?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > > > > > relango@salesforce.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > We are seeing a problem that we we try to send messages to
> > new
> > > > > topic
> > > > > > it
> > > > > > > > > fails kafka.common.LeaderNotAvailableException. But usually
> > > this
> > > > > > > problem
> > > > > > > > > will be transient and if we re-send messages to same topic
> > will
> > > > > work.
> > > > > > > But
> > > > > > > > > now we tried rending message to same topic several time,
> but
> > > > still
> > > > > > > fails
> > > > > > > > > with same error:
> > > > > > > > >
> > > > > > > > > In the server log I see ] Auto creation of topic test-sjl2
> > > with 8
> > > > > > > > > partitions and replication factor 3 is successful!. But
> > > > listTopics
> > > > > > > > command
> > > > > > > > > shows leader "none" like below:
> > > > > > > > >
> > > > > > > > > topic: test-sjl2 partition: 0 leader: none
> > > replicas:
> > > > > > 2,4,3
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 1 leader: none
> > > replicas:
> > > > > > 3,2,4
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 2 leader: none
> > > replicas:
> > > > > > 4,3,2
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 3 leader: none
> > > replicas:
> > > > > > 2,3,4
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 4 leader: none
> > > replicas:
> > > > > > 3,4,2
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 5 leader: none
> > > replicas:
> > > > > > 4,2,3
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 6 leader: none
> > > replicas:
> > > > > > 2,4,3
> > > > > > > > > isr:
> > > > > > > > > topic: test-sjl2 partition: 7 leader: none
> > > replicas:
> > > > > > 3,2,4
> > > > > > > > > isr:
> > > > > > > > >
> > > > > > > > > I also see following NotLeaderForPatritionExcetion and
> > > > > > > ZookeeperExcetion
> > > > > > > > in
> > > > > > > > > logs
> > > > > > > > >
> > > > > > > > > kafka.common.NotLeaderForPartitionException
> > > > > > > > > at
> > > > > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > > > > > > Source)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > > > > at
> > > > > > > >
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > > > > > > at java.lang.Class.newInstance(Class.java:308)
> > > > > > > > > at
> > > > > > > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > > > > > > at
> > > > > > > >
> > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > > > > > > at
> > > > > > > >
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > > > "leader_epoch":2,
> > > > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > > > KeeperErrorCode =
> > > > > > > > > BadVersion for
> /brokers/topics/FunnelProto/partitions/4/state
> > > > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > > > "leader_epoch":2,
> > > > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > > > KeeperErrorCode =
> > > > > > > > > BadVersion for
> /brokers/topics/FunnelProto/partitions/4/state
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Any clues on what could be problem.. ?
> > > > > > > > >
> > > > > > > > > Any for your help.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks,
> > > > > > > > > Raja.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Raja.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>
Re: Leader doesn't get assigned for new topics
Posted by Rajasekar Elango <re...@salesforce.com>.
>From the output of StateChangeLogMerger tool, I see only this error
repeated;
[2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
for partition [FunnelProto,0] (kafka.server.KafkaApis)
On the state-change.log itself, I see this error:
[2013-09-18 14:22:48,954] ERROR Conditional update of path
/brokers/topics/test-1379439240191/partitions/2/state with data {
"controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
"version":1 } and expected version 8 fai
led due to org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/brokers/topics/test-1379439240191/partitions/2/state (kafka.utils.ZkUtils$)
Do you know reason for above error..? Also this problem seem to be
intermittent, it started working now without any changes. I will continue
to monitor.
Thanks,
Raja.
On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Raja,
>
> Could you run the StateChangeLogMerger tool and give it one topic-partition
> that has the above mentioned problem. This tool is documented here -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> .
>
> Let me know if you run into any issues while using it.
>
> Thanks,
> Neha
>
>
> On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> <re...@salesforce.com>wrote:
>
> > Neha/Jun,
> >
> > The same problem started happening again although now our zookeeper
> cluster
> > is configured correctly. The produce always failed with
> > LeaderNotAvailableException and list topics shows topic is created with
> > leader "none". In the controller and stage-change log, I am seeing lot of
> > these failures..
> >
> >
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client on partition [FunnelProto,6] failed
> > due to Partition [FunnelProto,6] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client on partition [internal_metrics,3]
> > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client on partition [FunnelProto,0] failed
> > due to Partition [FunnelProto,0] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client on partition [jmx,3] failed due to
> > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client on partition [FunnelProto,5] failed
> > due to Partition [FunnelProto,5] doesn't exist on 2
> > (kafka.server.KafkaApis)
> >
> >
> > When I ran listTopics command for one of above topic, all partitions are
> > under replicated (we have replication factor set to 3). Any clues on what
> > could be issue and how can we get it back to working?
> >
> > Thanks,
> > Raja.
> >
> >
> >
> > On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Ah ok. Thanks for sharing that.
> > >
> > >
> > >
> > > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> > relango@salesforce.com
> > > >wrote:
> > >
> > > > We have 3 zookeeper node in the cluster with a hardware load
> balancer .
> > > In
> > > > one of the zookeeper, we did not configure ensemble correctly
> (server.n
> > > > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster,
> > one
> > > > node in other cluster. The load balancer is randomly hitting one of 2
> > > > zookeepers in two different cluster.
> > > >
> > > > Thanks,
> > > > Raja.
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Just curious to know, what was the misconfiguration?
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > > > > <re...@salesforce.com>wrote:
> > > > >
> > > > > > Thanks Neha and Jun, It turned out to be miss configuration in
> our
> > > > > > zookeeper cluster. After correcting it everything looks good.
> > > > > >
> > > > > > Thanks,
> > > > > > Raja.
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Any error in the controller and the state-change log? Are
> brokers
> > > > 2,3,4
> > > > > > > alive?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > > > > relango@salesforce.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > We are seeing a problem that we we try to send messages to
> new
> > > > topic
> > > > > it
> > > > > > > > fails kafka.common.LeaderNotAvailableException. But usually
> > this
> > > > > > problem
> > > > > > > > will be transient and if we re-send messages to same topic
> will
> > > > work.
> > > > > > But
> > > > > > > > now we tried rending message to same topic several time, but
> > > still
> > > > > > fails
> > > > > > > > with same error:
> > > > > > > >
> > > > > > > > In the server log I see ] Auto creation of topic test-sjl2
> > with 8
> > > > > > > > partitions and replication factor 3 is successful!. But
> > > listTopics
> > > > > > > command
> > > > > > > > shows leader "none" like below:
> > > > > > > >
> > > > > > > > topic: test-sjl2 partition: 0 leader: none
> > replicas:
> > > > > 2,4,3
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 1 leader: none
> > replicas:
> > > > > 3,2,4
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 2 leader: none
> > replicas:
> > > > > 4,3,2
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 3 leader: none
> > replicas:
> > > > > 2,3,4
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 4 leader: none
> > replicas:
> > > > > 3,4,2
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 5 leader: none
> > replicas:
> > > > > 4,2,3
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 6 leader: none
> > replicas:
> > > > > 2,4,3
> > > > > > > > isr:
> > > > > > > > topic: test-sjl2 partition: 7 leader: none
> > replicas:
> > > > > 3,2,4
> > > > > > > > isr:
> > > > > > > >
> > > > > > > > I also see following NotLeaderForPatritionExcetion and
> > > > > > ZookeeperExcetion
> > > > > > > in
> > > > > > > > logs
> > > > > > > >
> > > > > > > > kafka.common.NotLeaderForPartitionException
> > > > > > > > at
> > > > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > > > > > Source)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > > > at
> > > > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > > > > > at java.lang.Class.newInstance(Class.java:308)
> > > > > > > > at
> > > > > > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > > > > > at
> > > > > > >
> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > > > > > at
> > > > > > > >
> > > > >
> > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > >
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > > > > > at
> > > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > > "leader_epoch":2,
> > > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > > KeeperErrorCode =
> > > > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > > "leader_epoch":2,
> > > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > > KeeperErrorCode =
> > > > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > > > >
> > > > > > > >
> > > > > > > > Any clues on what could be problem.. ?
> > > > > > > >
> > > > > > > > Any for your help.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks,
> > > > > > > > Raja.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Raja.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Raja.
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Raja.
> >
>
--
Thanks,
Raja.
Re: Leader doesn't get assigned for new topics
Posted by Neha Narkhede <ne...@gmail.com>.
Raja,
Could you run the StateChangeLogMerger tool and give it one topic-partition
that has the above mentioned problem. This tool is documented here -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
.
Let me know if you run into any issues while using it.
Thanks,
Neha
On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
<re...@salesforce.com>wrote:
> Neha/Jun,
>
> The same problem started happening again although now our zookeeper cluster
> is configured correctly. The produce always failed with
> LeaderNotAvailableException and list topics shows topic is created with
> leader "none". In the controller and stage-change log, I am seeing lot of
> these failures..
>
>
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client on partition [FunnelProto,6] failed
> due to Partition [FunnelProto,6] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client on partition [internal_metrics,3]
> failed due to Partition [internal_metrics,3] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client on partition [FunnelProto,0] failed
> due to Partition [FunnelProto,0] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client on partition [jmx,3] failed due to
> Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client on partition [FunnelProto,5] failed
> due to Partition [FunnelProto,5] doesn't exist on 2
> (kafka.server.KafkaApis)
>
>
> When I ran listTopics command for one of above topic, all partitions are
> under replicated (we have replication factor set to 3). Any clues on what
> could be issue and how can we get it back to working?
>
> Thanks,
> Raja.
>
>
>
> On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Ah ok. Thanks for sharing that.
> >
> >
> >
> > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> relango@salesforce.com
> > >wrote:
> >
> > > We have 3 zookeeper node in the cluster with a hardware load balancer .
> > In
> > > one of the zookeeper, we did not configure ensemble correctly (server.n
> > > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster,
> one
> > > node in other cluster. The load balancer is randomly hitting one of 2
> > > zookeepers in two different cluster.
> > >
> > > Thanks,
> > > Raja.
> > >
> > >
> > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Just curious to know, what was the misconfiguration?
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > > > <re...@salesforce.com>wrote:
> > > >
> > > > > Thanks Neha and Jun, It turned out to be miss configuration in our
> > > > > zookeeper cluster. After correcting it everything looks good.
> > > > >
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Any error in the controller and the state-change log? Are brokers
> > > 2,3,4
> > > > > > alive?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > > > relango@salesforce.com
> > > > > > >wrote:
> > > > > >
> > > > > > > We are seeing a problem that we we try to send messages to new
> > > topic
> > > > it
> > > > > > > fails kafka.common.LeaderNotAvailableException. But usually
> this
> > > > > problem
> > > > > > > will be transient and if we re-send messages to same topic will
> > > work.
> > > > > But
> > > > > > > now we tried rending message to same topic several time, but
> > still
> > > > > fails
> > > > > > > with same error:
> > > > > > >
> > > > > > > In the server log I see ] Auto creation of topic test-sjl2
> with 8
> > > > > > > partitions and replication factor 3 is successful!. But
> > listTopics
> > > > > > command
> > > > > > > shows leader "none" like below:
> > > > > > >
> > > > > > > topic: test-sjl2 partition: 0 leader: none
> replicas:
> > > > 2,4,3
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 1 leader: none
> replicas:
> > > > 3,2,4
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 2 leader: none
> replicas:
> > > > 4,3,2
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 3 leader: none
> replicas:
> > > > 2,3,4
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 4 leader: none
> replicas:
> > > > 3,4,2
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 5 leader: none
> replicas:
> > > > 4,2,3
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 6 leader: none
> replicas:
> > > > 2,4,3
> > > > > > > isr:
> > > > > > > topic: test-sjl2 partition: 7 leader: none
> replicas:
> > > > 3,2,4
> > > > > > > isr:
> > > > > > >
> > > > > > > I also see following NotLeaderForPatritionExcetion and
> > > > > ZookeeperExcetion
> > > > > > in
> > > > > > > logs
> > > > > > >
> > > > > > > kafka.common.NotLeaderForPartitionException
> > > > > > > at
> > > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > > > > Source)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > > at
> > > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > > > > at java.lang.Class.newInstance(Class.java:308)
> > > > > > > at
> > > > > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > > > > at
> > > > > > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > > > > at
> > > > > > >
> > > >
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > > > > at
> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > "leader_epoch":2,
> > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > KeeperErrorCode =
> > > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> > "leader_epoch":2,
> > > > > > > "version":1 } and expected version 14 failed due to
> > > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > > KeeperErrorCode =
> > > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > > >
> > > > > > >
> > > > > > > Any clues on what could be problem.. ?
> > > > > > >
> > > > > > > Any for your help.
> > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Raja.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>
Re: Leader doesn't get assigned for new topics
Posted by Rajasekar Elango <re...@salesforce.com>.
Neha/Jun,
The same problem started happening again although now our zookeeper cluster
is configured correctly. The produce always failed with
LeaderNotAvailableException and list topics shows topic is created with
leader "none". In the controller and stage-change log, I am seeing lot of
these failures..
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [FunnelProto,6] failed
due to Partition [FunnelProto,6] doesn't exist on 2 (kafka.server.KafkaApis)
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [internal_metrics,3]
failed due to Partition [internal_metrics,3] doesn't exist on 2
(kafka.server.KafkaApis)
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [FunnelProto,0] failed
due to Partition [FunnelProto,0] doesn't exist on 2 (kafka.server.KafkaApis)
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [jmx,3] failed due to
Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client on partition [FunnelProto,5] failed
due to Partition [FunnelProto,5] doesn't exist on 2 (kafka.server.KafkaApis)
When I ran listTopics command for one of above topic, all partitions are
under replicated (we have replication factor set to 3). Any clues on what
could be issue and how can we get it back to working?
Thanks,
Raja.
On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Ah ok. Thanks for sharing that.
>
>
>
> On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <relango@salesforce.com
> >wrote:
>
> > We have 3 zookeeper node in the cluster with a hardware load balancer .
> In
> > one of the zookeeper, we did not configure ensemble correctly (server.n
> > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster, one
> > node in other cluster. The load balancer is randomly hitting one of 2
> > zookeepers in two different cluster.
> >
> > Thanks,
> > Raja.
> >
> >
> > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Just curious to know, what was the misconfiguration?
> > >
> > >
> > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > > <re...@salesforce.com>wrote:
> > >
> > > > Thanks Neha and Jun, It turned out to be miss configuration in our
> > > > zookeeper cluster. After correcting it everything looks good.
> > > >
> > > > Thanks,
> > > > Raja.
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Any error in the controller and the state-change log? Are brokers
> > 2,3,4
> > > > > alive?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > > relango@salesforce.com
> > > > > >wrote:
> > > > >
> > > > > > We are seeing a problem that we we try to send messages to new
> > topic
> > > it
> > > > > > fails kafka.common.LeaderNotAvailableException. But usually this
> > > > problem
> > > > > > will be transient and if we re-send messages to same topic will
> > work.
> > > > But
> > > > > > now we tried rending message to same topic several time, but
> still
> > > > fails
> > > > > > with same error:
> > > > > >
> > > > > > In the server log I see ] Auto creation of topic test-sjl2 with 8
> > > > > > partitions and replication factor 3 is successful!. But
> listTopics
> > > > > command
> > > > > > shows leader "none" like below:
> > > > > >
> > > > > > topic: test-sjl2 partition: 0 leader: none replicas:
> > > 2,4,3
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 1 leader: none replicas:
> > > 3,2,4
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 2 leader: none replicas:
> > > 4,3,2
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 3 leader: none replicas:
> > > 2,3,4
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 4 leader: none replicas:
> > > 3,4,2
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 5 leader: none replicas:
> > > 4,2,3
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 6 leader: none replicas:
> > > 2,4,3
> > > > > > isr:
> > > > > > topic: test-sjl2 partition: 7 leader: none replicas:
> > > 3,2,4
> > > > > > isr:
> > > > > >
> > > > > > I also see following NotLeaderForPatritionExcetion and
> > > > ZookeeperExcetion
> > > > > in
> > > > > > logs
> > > > > >
> > > > > > kafka.common.NotLeaderForPartitionException
> > > > > > at
> > > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > > > Source)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > > at
> > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > > > at java.lang.Class.newInstance(Class.java:308)
> > > > > > at
> > > > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > > > at
> > > > > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > > > at
> > > > > >
> > > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > > > at
> > > > > >
> > > >
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > > > at
> > > > > >
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > > > at
> > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> "leader_epoch":2,
> > > > > > "version":1 } and expected version 14 failed due to
> > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > KeeperErrorCode =
> > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2,
> "leader_epoch":2,
> > > > > > "version":1 } and expected version 14 failed due to
> > > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > > KeeperErrorCode =
> > > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > >
> > > > > >
> > > > > > Any clues on what could be problem.. ?
> > > > > >
> > > > > > Any for your help.
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Raja.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Raja.
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Raja.
> >
>
--
Thanks,
Raja.
Re: Leader doesn't get assigned for new topics
Posted by Neha Narkhede <ne...@gmail.com>.
Ah ok. Thanks for sharing that.
On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <re...@salesforce.com>wrote:
> We have 3 zookeeper node in the cluster with a hardware load balancer . In
> one of the zookeeper, we did not configure ensemble correctly (server.n
> property in zoo.cfg) . So it ended up as like 2 nodes in one cluster, one
> node in other cluster. The load balancer is randomly hitting one of 2
> zookeepers in two different cluster.
>
> Thanks,
> Raja.
>
>
> On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Just curious to know, what was the misconfiguration?
> >
> >
> > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > <re...@salesforce.com>wrote:
> >
> > > Thanks Neha and Jun, It turned out to be miss configuration in our
> > > zookeeper cluster. After correcting it everything looks good.
> > >
> > > Thanks,
> > > Raja.
> > >
> > >
> > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Any error in the controller and the state-change log? Are brokers
> 2,3,4
> > > > alive?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > relango@salesforce.com
> > > > >wrote:
> > > >
> > > > > We are seeing a problem that we we try to send messages to new
> topic
> > it
> > > > > fails kafka.common.LeaderNotAvailableException. But usually this
> > > problem
> > > > > will be transient and if we re-send messages to same topic will
> work.
> > > But
> > > > > now we tried rending message to same topic several time, but still
> > > fails
> > > > > with same error:
> > > > >
> > > > > In the server log I see ] Auto creation of topic test-sjl2 with 8
> > > > > partitions and replication factor 3 is successful!. But listTopics
> > > > command
> > > > > shows leader "none" like below:
> > > > >
> > > > > topic: test-sjl2 partition: 0 leader: none replicas:
> > 2,4,3
> > > > > isr:
> > > > > topic: test-sjl2 partition: 1 leader: none replicas:
> > 3,2,4
> > > > > isr:
> > > > > topic: test-sjl2 partition: 2 leader: none replicas:
> > 4,3,2
> > > > > isr:
> > > > > topic: test-sjl2 partition: 3 leader: none replicas:
> > 2,3,4
> > > > > isr:
> > > > > topic: test-sjl2 partition: 4 leader: none replicas:
> > 3,4,2
> > > > > isr:
> > > > > topic: test-sjl2 partition: 5 leader: none replicas:
> > 4,2,3
> > > > > isr:
> > > > > topic: test-sjl2 partition: 6 leader: none replicas:
> > 2,4,3
> > > > > isr:
> > > > > topic: test-sjl2 partition: 7 leader: none replicas:
> > 3,2,4
> > > > > isr:
> > > > >
> > > > > I also see following NotLeaderForPatritionExcetion and
> > > ZookeeperExcetion
> > > > in
> > > > > logs
> > > > >
> > > > > kafka.common.NotLeaderForPartitionException
> > > > > at
> > > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > > Source)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > > at
> > > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > > at java.lang.Class.newInstance(Class.java:308)
> > > > > at
> > > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > > at
> > > > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > > at
> > > > >
> > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > > at
> > > > >
> > >
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > > at
> > > > >
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > > at
> > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > > > "version":1 } and expected version 14 failed due to
> > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > KeeperErrorCode =
> > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > > > "version":1 } and expected version 14 failed due to
> > > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > KeeperErrorCode =
> > > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > >
> > > > >
> > > > > Any clues on what could be problem.. ?
> > > > >
> > > > > Any for your help.
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>
Re: Leader doesn't get assigned for new topics
Posted by Rajasekar Elango <re...@salesforce.com>.
We have 3 zookeeper node in the cluster with a hardware load balancer . In
one of the zookeeper, we did not configure ensemble correctly (server.n
property in zoo.cfg) . So it ended up as like 2 nodes in one cluster, one
node in other cluster. The load balancer is randomly hitting one of 2
zookeepers in two different cluster.
Thanks,
Raja.
On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Just curious to know, what was the misconfiguration?
>
>
> On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> <re...@salesforce.com>wrote:
>
> > Thanks Neha and Jun, It turned out to be miss configuration in our
> > zookeeper cluster. After correcting it everything looks good.
> >
> > Thanks,
> > Raja.
> >
> >
> > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Any error in the controller and the state-change log? Are brokers 2,3,4
> > > alive?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > relango@salesforce.com
> > > >wrote:
> > >
> > > > We are seeing a problem that we we try to send messages to new topic
> it
> > > > fails kafka.common.LeaderNotAvailableException. But usually this
> > problem
> > > > will be transient and if we re-send messages to same topic will work.
> > But
> > > > now we tried rending message to same topic several time, but still
> > fails
> > > > with same error:
> > > >
> > > > In the server log I see ] Auto creation of topic test-sjl2 with 8
> > > > partitions and replication factor 3 is successful!. But listTopics
> > > command
> > > > shows leader "none" like below:
> > > >
> > > > topic: test-sjl2 partition: 0 leader: none replicas:
> 2,4,3
> > > > isr:
> > > > topic: test-sjl2 partition: 1 leader: none replicas:
> 3,2,4
> > > > isr:
> > > > topic: test-sjl2 partition: 2 leader: none replicas:
> 4,3,2
> > > > isr:
> > > > topic: test-sjl2 partition: 3 leader: none replicas:
> 2,3,4
> > > > isr:
> > > > topic: test-sjl2 partition: 4 leader: none replicas:
> 3,4,2
> > > > isr:
> > > > topic: test-sjl2 partition: 5 leader: none replicas:
> 4,2,3
> > > > isr:
> > > > topic: test-sjl2 partition: 6 leader: none replicas:
> 2,4,3
> > > > isr:
> > > > topic: test-sjl2 partition: 7 leader: none replicas:
> 3,2,4
> > > > isr:
> > > >
> > > > I also see following NotLeaderForPatritionExcetion and
> > ZookeeperExcetion
> > > in
> > > > logs
> > > >
> > > > kafka.common.NotLeaderForPartitionException
> > > > at
> > sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > > Source)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > > at
> > > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > > at java.lang.Class.newInstance0(Class.java:355)
> > > > at java.lang.Class.newInstance(Class.java:308)
> > > > at
> > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > > at
> > > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > > at
> > > >
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > at
> > > >
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > > at
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > > at
> > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > > "version":1 } and expected version 14 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode =
> > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > > "version":1 } and expected version 14 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode =
> > > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > >
> > > >
> > > > Any clues on what could be problem.. ?
> > > >
> > > > Any for your help.
> > > >
> > > > --
> > > > Thanks,
> > > > Raja.
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Raja.
> >
>
--
Thanks,
Raja.
Re: Leader doesn't get assigned for new topics
Posted by Neha Narkhede <ne...@gmail.com>.
Just curious to know, what was the misconfiguration?
On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
<re...@salesforce.com>wrote:
> Thanks Neha and Jun, It turned out to be miss configuration in our
> zookeeper cluster. After correcting it everything looks good.
>
> Thanks,
> Raja.
>
>
> On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Any error in the controller and the state-change log? Are brokers 2,3,4
> > alive?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> relango@salesforce.com
> > >wrote:
> >
> > > We are seeing a problem that we we try to send messages to new topic it
> > > fails kafka.common.LeaderNotAvailableException. But usually this
> problem
> > > will be transient and if we re-send messages to same topic will work.
> But
> > > now we tried rending message to same topic several time, but still
> fails
> > > with same error:
> > >
> > > In the server log I see ] Auto creation of topic test-sjl2 with 8
> > > partitions and replication factor 3 is successful!. But listTopics
> > command
> > > shows leader "none" like below:
> > >
> > > topic: test-sjl2 partition: 0 leader: none replicas: 2,4,3
> > > isr:
> > > topic: test-sjl2 partition: 1 leader: none replicas: 3,2,4
> > > isr:
> > > topic: test-sjl2 partition: 2 leader: none replicas: 4,3,2
> > > isr:
> > > topic: test-sjl2 partition: 3 leader: none replicas: 2,3,4
> > > isr:
> > > topic: test-sjl2 partition: 4 leader: none replicas: 3,4,2
> > > isr:
> > > topic: test-sjl2 partition: 5 leader: none replicas: 4,2,3
> > > isr:
> > > topic: test-sjl2 partition: 6 leader: none replicas: 2,4,3
> > > isr:
> > > topic: test-sjl2 partition: 7 leader: none replicas: 3,2,4
> > > isr:
> > >
> > > I also see following NotLeaderForPatritionExcetion and
> ZookeeperExcetion
> > in
> > > logs
> > >
> > > kafka.common.NotLeaderForPartitionException
> > > at
> sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > > Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at java.lang.Class.newInstance0(Class.java:355)
> > > at java.lang.Class.newInstance(Class.java:308)
> > > at
> kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > at
> > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > > at
> > > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > at
> > >
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > > at
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > "version":1 } and expected version 14 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode =
> > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > > (kafka.utils.ZkUtils$) - Conditional update of path
> > > /brokers/topics/FunnelProto/partitions/4/state with data {
> > > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > > "version":1 } and expected version 14 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode =
> > > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > >
> > >
> > > Any clues on what could be problem.. ?
> > >
> > > Any for your help.
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>
Re: Leader doesn't get assigned for new topics
Posted by Rajasekar Elango <re...@salesforce.com>.
Thanks Neha and Jun, It turned out to be miss configuration in our
zookeeper cluster. After correcting it everything looks good.
Thanks,
Raja.
On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <ju...@gmail.com> wrote:
> Any error in the controller and the state-change log? Are brokers 2,3,4
> alive?
>
> Thanks,
>
> Jun
>
>
> On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <relango@salesforce.com
> >wrote:
>
> > We are seeing a problem that we we try to send messages to new topic it
> > fails kafka.common.LeaderNotAvailableException. But usually this problem
> > will be transient and if we re-send messages to same topic will work. But
> > now we tried rending message to same topic several time, but still fails
> > with same error:
> >
> > In the server log I see ] Auto creation of topic test-sjl2 with 8
> > partitions and replication factor 3 is successful!. But listTopics
> command
> > shows leader "none" like below:
> >
> > topic: test-sjl2 partition: 0 leader: none replicas: 2,4,3
> > isr:
> > topic: test-sjl2 partition: 1 leader: none replicas: 3,2,4
> > isr:
> > topic: test-sjl2 partition: 2 leader: none replicas: 4,3,2
> > isr:
> > topic: test-sjl2 partition: 3 leader: none replicas: 2,3,4
> > isr:
> > topic: test-sjl2 partition: 4 leader: none replicas: 3,4,2
> > isr:
> > topic: test-sjl2 partition: 5 leader: none replicas: 4,2,3
> > isr:
> > topic: test-sjl2 partition: 6 leader: none replicas: 2,4,3
> > isr:
> > topic: test-sjl2 partition: 7 leader: none replicas: 3,2,4
> > isr:
> >
> > I also see following NotLeaderForPatritionExcetion and ZookeeperExcetion
> in
> > logs
> >
> > kafka.common.NotLeaderForPartitionException
> > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> > Source)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at java.lang.Class.newInstance0(Class.java:355)
> > at java.lang.Class.newInstance(Class.java:308)
> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> > at kafka.utils.Logging$class.warn(Logging.scala:88)
> > at
> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> > at
> >
> >
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> > at
> > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > at
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > at
> >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> > at
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> > at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > (kafka.utils.ZkUtils$) - Conditional update of path
> > /brokers/topics/FunnelProto/partitions/4/state with data {
> > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > "version":1 } and expected version 14 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode =
> > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> > 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> > (kafka.utils.ZkUtils$) - Conditional update of path
> > /brokers/topics/FunnelProto/partitions/4/state with data {
> > "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> > "version":1 } and expected version 14 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode =
> > BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> >
> >
> > Any clues on what could be problem.. ?
> >
> > Any for your help.
> >
> > --
> > Thanks,
> > Raja.
> >
>
--
Thanks,
Raja.
Re: Leader doesn't get assigned for new topics
Posted by Jun Rao <ju...@gmail.com>.
Any error in the controller and the state-change log? Are brokers 2,3,4
alive?
Thanks,
Jun
On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <re...@salesforce.com>wrote:
> We are seeing a problem that we we try to send messages to new topic it
> fails kafka.common.LeaderNotAvailableException. But usually this problem
> will be transient and if we re-send messages to same topic will work. But
> now we tried rending message to same topic several time, but still fails
> with same error:
>
> In the server log I see ] Auto creation of topic test-sjl2 with 8
> partitions and replication factor 3 is successful!. But listTopics command
> shows leader "none" like below:
>
> topic: test-sjl2 partition: 0 leader: none replicas: 2,4,3
> isr:
> topic: test-sjl2 partition: 1 leader: none replicas: 3,2,4
> isr:
> topic: test-sjl2 partition: 2 leader: none replicas: 4,3,2
> isr:
> topic: test-sjl2 partition: 3 leader: none replicas: 2,3,4
> isr:
> topic: test-sjl2 partition: 4 leader: none replicas: 3,4,2
> isr:
> topic: test-sjl2 partition: 5 leader: none replicas: 4,2,3
> isr:
> topic: test-sjl2 partition: 6 leader: none replicas: 2,4,3
> isr:
> topic: test-sjl2 partition: 7 leader: none replicas: 3,2,4
> isr:
>
> I also see following NotLeaderForPatritionExcetion and ZookeeperExcetion in
> logs
>
> kafka.common.NotLeaderForPartitionException
> at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown
> Source)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> at kafka.utils.Logging$class.warn(Logging.scala:88)
> at kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> at
>
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> at
>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> (kafka.utils.ZkUtils$) - Conditional update of path
> /brokers/topics/FunnelProto/partitions/4/state with data {
> "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> "version":1 } and expected version 14 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/FunnelProto/partitions/4/state
> 2013-09-12 23:54:10,838 [kafka-request-handler-2] ERROR
> (kafka.utils.ZkUtils$) - Conditional update of path
> /brokers/topics/FunnelProto/partitions/4/state with data {
> "controller_epoch":3, "isr":[ 2, 5 ], "leader":2, "leader_epoch":2,
> "version":1 } and expected version 14 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/FunnelProto/partitions/4/state
>
>
> Any clues on what could be problem.. ?
>
> Any for your help.
>
> --
> Thanks,
> Raja.
>