You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Prakash Gowri Shankor <pr...@gmail.com> on 2014/06/10 20:04:32 UTC
LeaderNotAvailableException in 0.8.1.1
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted (kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted (kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted (kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48] (kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 ->
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
Is this what you want from kafka-topics ? I took this script dump now when
the exception is occuring.
./kafka-topics.sh --describe test2 --zookeeper localhost:2181
Topic:test2 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test2 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test2 Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: test2 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
On Tue, Jun 10, 2014 at 9:22 PM, Jun Rao <ju...@gmail.com> wrote:
> Could you use the kafka-topic command to describe test2 and see if the
> leader is available?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> prakash.shankor@gmail.com> wrote:
>
> > Hi,
> >
> > I am running a cluster with a single broker, the performance producer
> > script and 3 consumers.
> > On a fresh start of the cluster , the producer throws this exception.
> > I was able to run this cluster successfully on the same topic ( test2 )
> > successfully the first time.
> >
> > The solution( from stackoverflow ) seems to be to delete the topic data
> in
> > the broker and in zookeeper. This doesnt seem to be a viable production
> > solution to me. Is there a way to solve this without losing topic data ?
> >
> > Also does the incidence of this problem decrease if I run more
> > brokers/servers ?
> >
> > I see logs like these in the server.log
> >
> > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,0] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,1] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,2] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > The exception trace is:
> >
> > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > (kafka.producer.async.ProducerSendThread)
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> >
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> >
> > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> >
> > at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
> > with correlation ids in [41,48]
> (kafka.producer.async.DefaultEventHandler)
> >
> > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > [{TopicMetadata for topic test2 ->
> >
> > No partition metadata for topic test2 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> >
> > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > (kafka.producer.async.ProducerSendThread)
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> >
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> >
> > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> >
> > at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Jun Rao <ju...@gmail.com>.
Could you use the kafka-topic command to describe test2 and see if the
leader is available?
Thanks,
Jun
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
prakash.shankor@gmail.com> wrote:
> Hi,
>
> I am running a cluster with a single broker, the performance producer
> script and 3 consumers.
> On a fresh start of the cluster , the producer throws this exception.
> I was able to run this cluster successfully on the same topic ( test2 )
> successfully the first time.
>
> The solution( from stackoverflow ) seems to be to delete the topic data in
> the broker and in zookeeper. This doesnt seem to be a viable production
> solution to me. Is there a way to solve this without losing topic data ?
>
> Also does the incidence of this problem decrease if I run more
> brokers/servers ?
>
> I see logs like these in the server.log
>
> [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,0] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,1] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,2] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> The exception trace is:
>
> [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> (kafka.producer.async.ProducerSendThread)
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
>
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
>
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
> [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
> with correlation ids in [41,48] (kafka.producer.async.DefaultEventHandler)
>
> [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> [{TopicMetadata for topic test2 ->
>
> No partition metadata for topic test2 due to
> kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
>
> [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> (kafka.producer.async.ProducerSendThread)
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
>
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
>
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Guozhang Wang <wa...@gmail.com>.
Any exceptions you saw on the controller log?
On Tue, Jun 10, 2014 at 2:23 PM, Prakash Gowri Shankor <
prakash.shankor@gmail.com> wrote:
> No i have not used the delete topic feature. I have been manually deleting
> the topics from zookeeper and removing the topic from the kafka and zk
> logs.
> I've experimented a bit more. It seems like this occurs when I have a
> single broker running. When i restart with 2 brokers, the problem goes
> away.
>
>
> On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > Did you use the delete topic command?
> >
> > That was an experimental feature in the 0.8.1 release with several
> > bugs. The fixes are all on trunk, but those fixes did not make it into
> > 0.8.1.1 - except for a config option to disable delete-topic support
> > on the broker.
> >
> > Joel
> >
> > On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
> > > From the moment it starts occuring, it is persistent. Restarts dont
> seem
> > to
> > > make it go away. The only thing that makes it go away is following the
> > > steps listed in this stackoverflow thread.
> > >
> > >
> >
> http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
> > >
> > >
> > >
> > > On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hello Prakash,
> > > >
> > > > Is this exception transient or persistent on broker startup?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> > > > prakash.shankor@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am running a cluster with a single broker, the performance
> producer
> > > > > script and 3 consumers.
> > > > > On a fresh start of the cluster , the producer throws this
> exception.
> > > > > I was able to run this cluster successfully on the same topic (
> > test2 )
> > > > > successfully the first time.
> > > > >
> > > > > The solution( from stackoverflow ) seems to be to delete the topic
> > data
> > > > in
> > > > > the broker and in zookeeper. This doesnt seem to be a viable
> > production
> > > > > solution to me. Is there a way to solve this without losing topic
> > data ?
> > > > >
> > > > > Also does the incidence of this problem decrease if I run more
> > > > > brokers/servers ?
> > > > >
> > > > > I see logs like these in the server.log
> > > > >
> > > > > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,0] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,1] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,2] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > The exception trace is:
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
> events
> > > > > (kafka.producer.async.ProducerSendThread)
> > > > >
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after
> > > > 3
> > > > > tries.
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > >
> > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > >
> > > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
> > test2
> > > > > with correlation ids in [41,48]
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > >
> > > > > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test2 ->
> > > > >
> > > > > No partition metadata for topic test2 due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
> events
> > > > > (kafka.producer.async.ProducerSendThread)
> > > > >
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after
> > > > 3
> > > > > tries.
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > >
> > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > >
> > > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> >
> >
>
--
-- Guozhang
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
yes, I gave it several minutes.
On Sat, Jun 14, 2014 at 2:18 PM, Michael G. Noll <mi...@michael-noll.com>
wrote:
> Have you given Kafka some time to re-elect a new leader for the
> "missing" partition when you re-try steps 1-5?
>
> See here:
> > If you do, you should be able to go through steps
> > 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
> > Kafka some time to re-elect the remaining, second broker as the new
> > leader for the first broker's partitions though).
>
> Best,
> Michael
>
>
>
> On 06/12/2014 08:43 PM, Prakash Gowri Shankor wrote:
> > So if we go back to the 2 broker case, I tried your suggestion with
> > replication-factor 2
> >
> > ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper
> > localhost:2181 --replication-factor
> >
> > When i repeat steps 1-5 i still see the exception. When i go to step 8 (
> > back to 2 brokers ), I dont see it.
> > Here is my topic description:
> >
> > ./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181
> >
> > Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs:
> >
> > Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0
> >
> > Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
> >
> > Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0
> >
> >
> > On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll <
> > michael+storm@michael-noll.com> wrote:
> >
> >> In your second case (1-broker cluster and putting your laptop to sleep)
> >> these exceptions should be transient and disappear after a while.
> >>
> >> In the logs you should see ZK session expirations (hence the
> >> initial/transient exceptions, which in this case are expected and ok),
> >> followed by new ZK sessions being established.
> >>
> >> So this case is (should?) be very different from your case number 1.
> >>
> >> --Michael
> >>
> >>
> >>> On 11.06.2014, at 23:13, Prakash Gowri Shankor <
> >> prakash.shankor@gmail.com> wrote:
> >>>
> >>> Thanks for your response Michael.
> >>>
> >>> In step 3, I am actually stopping the entire cluster and restarting it
> >>> without the 2nd broker. But I see your point. When i look in
> >>> /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it
> >>> holds test2-1 ( ie 1st partition of test2 topic ).
> >>> For /tmp/kafka-logs ( which is the log dir for the first broker ) I see
> >> it
> >>> holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ).
> >>> So it would seem that kafka is missing the leader for partition 1 and
> >> hence
> >>> throwing the exception on the producer side.
> >>> Let me try your replication suggestion.
> >>>
> >>> While all of the above might explain the exception in the case of 2
> >>> brokers, there are still times when I see it with just a single broker.
> >>> In this case, I start from a normal working cluster with 1 broker only.
> >>> Then I either put my machine into sleep/hibernation. On wake, I do
> >> shutdown
> >>> the cluster ( for sanity ) and restart.
> >>> On restart, I start seeing this exception. In this case i only have one
> >>> broker. I still create the topic the way i described earlier.
> >>> I understand this is not the ideal production topology, but its
> annoying
> >> to
> >>> see it during development.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll <
> >> michael@michael-noll.com>
> >>> wrote:
> >>>
> >>>> Prakash,
> >>>>
> >>>> you are configure the topic with a replication factor of only 1, i.e.
> no
> >>>> additional replica beyond "the original one". This replication
> setting
> >>>> of 1 means that only one of the two brokers will ever host the
> (single)
> >>>> replica -- which is implied to also be the leader in-sync replica --
> of
> >>>> a given partition.
> >>>>
> >>>> In step 3 you are disabling one of the two brokers. Because this
> >>>> stopped broker is the only broker that hosts one or more of the 3
> >>>> partitions you configured (I can't tell which partition(s) it is, but
> >>>> you can find out by --describe'ing the topic), your Kafka cluster --
> >>>> which is now running in degraded state -- will miss the leader of
> those
> >>>> affected partitions. And because you set the replication factor to 1,
> >>>> the remaining, second broker will not and will never take over the
> >>>> leadership of those partitions from the stopped broker. Hence you
> will
> >>>> keep getting the LeaderNotAvailableException's until you restart the
> >>>> stopped broker in step 7.
> >>>>
> >>>> So to me it looks as if the behavior of Kafka is actually correct and
> as
> >>>> expected.
> >>>>
> >>>> If you want to "rectify" your test setup, try increasing the
> replication
> >>>> factor from 1 to 2. If you do, you should be able to go through steps
> >>>> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
> >>>> Kafka some time to re-elect the remaining, second broker as the new
> >>>> leader for the first broker's partitions though).
> >>>>
> >>>> Hope this helps,
> >>>> Michael
> >>>>
> >>>>
> >>>>
> >>>>> On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
> >>>>> yes,
> >>>>> here are the steps:
> >>>>>
> >>>>> Create topic as : ./kafka-topics.sh --topic test2 --create
> >>>> --partitions 3
> >>>>> --zookeeper localhost:2181 --replication-factor 1
> >>>>>
> >>>>> 1) Start cluster with 2 brokers, 3 consumers.
> >>>>> 2) Dont start any producer
> >>>>> 3) Shutdown cluster and disable one broker from starting
> >>>>> 4) restart cluster with 1 broker, 3 consumers
> >>>>> 5) Start producer and send messages. I see this exception
> >>>>> 6) Shutdown cluster.
> >>>>> 7) Enable 2nd broker.
> >>>>> 8) Restart cluster with 2 brokers, 3 consumer and the one producer
> and
> >>>> send
> >>>>> messages. Now I dont see the exception.
> >>>>
> >>>>
> >>
> >
>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
Have you given Kafka some time to re-elect a new leader for the
"missing" partition when you re-try steps 1-5?
See here:
> If you do, you should be able to go through steps
> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
> Kafka some time to re-elect the remaining, second broker as the new
> leader for the first broker's partitions though).
Best,
Michael
On 06/12/2014 08:43 PM, Prakash Gowri Shankor wrote:
> So if we go back to the 2 broker case, I tried your suggestion with
> replication-factor 2
>
> ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper
> localhost:2181 --replication-factor
>
> When i repeat steps 1-5 i still see the exception. When i go to step 8 (
> back to 2 brokers ), I dont see it.
> Here is my topic description:
>
> ./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181
>
> Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs:
>
> Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0
>
> Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
>
> Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0
>
>
> On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll <
> michael+storm@michael-noll.com> wrote:
>
>> In your second case (1-broker cluster and putting your laptop to sleep)
>> these exceptions should be transient and disappear after a while.
>>
>> In the logs you should see ZK session expirations (hence the
>> initial/transient exceptions, which in this case are expected and ok),
>> followed by new ZK sessions being established.
>>
>> So this case is (should?) be very different from your case number 1.
>>
>> --Michael
>>
>>
>>> On 11.06.2014, at 23:13, Prakash Gowri Shankor <
>> prakash.shankor@gmail.com> wrote:
>>>
>>> Thanks for your response Michael.
>>>
>>> In step 3, I am actually stopping the entire cluster and restarting it
>>> without the 2nd broker. But I see your point. When i look in
>>> /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it
>>> holds test2-1 ( ie 1st partition of test2 topic ).
>>> For /tmp/kafka-logs ( which is the log dir for the first broker ) I see
>> it
>>> holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ).
>>> So it would seem that kafka is missing the leader for partition 1 and
>> hence
>>> throwing the exception on the producer side.
>>> Let me try your replication suggestion.
>>>
>>> While all of the above might explain the exception in the case of 2
>>> brokers, there are still times when I see it with just a single broker.
>>> In this case, I start from a normal working cluster with 1 broker only.
>>> Then I either put my machine into sleep/hibernation. On wake, I do
>> shutdown
>>> the cluster ( for sanity ) and restart.
>>> On restart, I start seeing this exception. In this case i only have one
>>> broker. I still create the topic the way i described earlier.
>>> I understand this is not the ideal production topology, but its annoying
>> to
>>> see it during development.
>>>
>>> Thanks
>>>
>>>
>>> On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll <
>> michael@michael-noll.com>
>>> wrote:
>>>
>>>> Prakash,
>>>>
>>>> you are configure the topic with a replication factor of only 1, i.e. no
>>>> additional replica beyond "the original one". This replication setting
>>>> of 1 means that only one of the two brokers will ever host the (single)
>>>> replica -- which is implied to also be the leader in-sync replica -- of
>>>> a given partition.
>>>>
>>>> In step 3 you are disabling one of the two brokers. Because this
>>>> stopped broker is the only broker that hosts one or more of the 3
>>>> partitions you configured (I can't tell which partition(s) it is, but
>>>> you can find out by --describe'ing the topic), your Kafka cluster --
>>>> which is now running in degraded state -- will miss the leader of those
>>>> affected partitions. And because you set the replication factor to 1,
>>>> the remaining, second broker will not and will never take over the
>>>> leadership of those partitions from the stopped broker. Hence you will
>>>> keep getting the LeaderNotAvailableException's until you restart the
>>>> stopped broker in step 7.
>>>>
>>>> So to me it looks as if the behavior of Kafka is actually correct and as
>>>> expected.
>>>>
>>>> If you want to "rectify" your test setup, try increasing the replication
>>>> factor from 1 to 2. If you do, you should be able to go through steps
>>>> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
>>>> Kafka some time to re-elect the remaining, second broker as the new
>>>> leader for the first broker's partitions though).
>>>>
>>>> Hope this helps,
>>>> Michael
>>>>
>>>>
>>>>
>>>>> On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
>>>>> yes,
>>>>> here are the steps:
>>>>>
>>>>> Create topic as : ./kafka-topics.sh --topic test2 --create
>>>> --partitions 3
>>>>> --zookeeper localhost:2181 --replication-factor 1
>>>>>
>>>>> 1) Start cluster with 2 brokers, 3 consumers.
>>>>> 2) Dont start any producer
>>>>> 3) Shutdown cluster and disable one broker from starting
>>>>> 4) restart cluster with 1 broker, 3 consumers
>>>>> 5) Start producer and send messages. I see this exception
>>>>> 6) Shutdown cluster.
>>>>> 7) Enable 2nd broker.
>>>>> 8) Restart cluster with 2 brokers, 3 consumer and the one producer and
>>>> send
>>>>> messages. Now I dont see the exception.
>>>>
>>>>
>>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
So if we go back to the 2 broker case, I tried your suggestion with
replication-factor 2
./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper
localhost:2181 --replication-factor
When i repeat steps 1-5 i still see the exception. When i go to step 8 (
back to 2 brokers ), I dont see it.
Here is my topic description:
./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181
Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs:
Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0
Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0
On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll <
michael+storm@michael-noll.com> wrote:
> In your second case (1-broker cluster and putting your laptop to sleep)
> these exceptions should be transient and disappear after a while.
>
> In the logs you should see ZK session expirations (hence the
> initial/transient exceptions, which in this case are expected and ok),
> followed by new ZK sessions being established.
>
> So this case is (should?) be very different from your case number 1.
>
> --Michael
>
>
> > On 11.06.2014, at 23:13, Prakash Gowri Shankor <
> prakash.shankor@gmail.com> wrote:
> >
> > Thanks for your response Michael.
> >
> > In step 3, I am actually stopping the entire cluster and restarting it
> > without the 2nd broker. But I see your point. When i look in
> > /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it
> > holds test2-1 ( ie 1st partition of test2 topic ).
> > For /tmp/kafka-logs ( which is the log dir for the first broker ) I see
> it
> > holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ).
> > So it would seem that kafka is missing the leader for partition 1 and
> hence
> > throwing the exception on the producer side.
> > Let me try your replication suggestion.
> >
> > While all of the above might explain the exception in the case of 2
> > brokers, there are still times when I see it with just a single broker.
> > In this case, I start from a normal working cluster with 1 broker only.
> > Then I either put my machine into sleep/hibernation. On wake, I do
> shutdown
> > the cluster ( for sanity ) and restart.
> > On restart, I start seeing this exception. In this case i only have one
> > broker. I still create the topic the way i described earlier.
> > I understand this is not the ideal production topology, but its annoying
> to
> > see it during development.
> >
> > Thanks
> >
> >
> > On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll <
> michael@michael-noll.com>
> > wrote:
> >
> >> Prakash,
> >>
> >> you are configure the topic with a replication factor of only 1, i.e. no
> >> additional replica beyond "the original one". This replication setting
> >> of 1 means that only one of the two brokers will ever host the (single)
> >> replica -- which is implied to also be the leader in-sync replica -- of
> >> a given partition.
> >>
> >> In step 3 you are disabling one of the two brokers. Because this
> >> stopped broker is the only broker that hosts one or more of the 3
> >> partitions you configured (I can't tell which partition(s) it is, but
> >> you can find out by --describe'ing the topic), your Kafka cluster --
> >> which is now running in degraded state -- will miss the leader of those
> >> affected partitions. And because you set the replication factor to 1,
> >> the remaining, second broker will not and will never take over the
> >> leadership of those partitions from the stopped broker. Hence you will
> >> keep getting the LeaderNotAvailableException's until you restart the
> >> stopped broker in step 7.
> >>
> >> So to me it looks as if the behavior of Kafka is actually correct and as
> >> expected.
> >>
> >> If you want to "rectify" your test setup, try increasing the replication
> >> factor from 1 to 2. If you do, you should be able to go through steps
> >> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
> >> Kafka some time to re-elect the remaining, second broker as the new
> >> leader for the first broker's partitions though).
> >>
> >> Hope this helps,
> >> Michael
> >>
> >>
> >>
> >>> On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
> >>> yes,
> >>> here are the steps:
> >>>
> >>> Create topic as : ./kafka-topics.sh --topic test2 --create
> >> --partitions 3
> >>> --zookeeper localhost:2181 --replication-factor 1
> >>>
> >>> 1) Start cluster with 2 brokers, 3 consumers.
> >>> 2) Dont start any producer
> >>> 3) Shutdown cluster and disable one broker from starting
> >>> 4) restart cluster with 1 broker, 3 consumers
> >>> 5) Start producer and send messages. I see this exception
> >>> 6) Shutdown cluster.
> >>> 7) Enable 2nd broker.
> >>> 8) Restart cluster with 2 brokers, 3 consumer and the one producer and
> >> send
> >>> messages. Now I dont see the exception.
> >>
> >>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while.
In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessions being established.
So this case is (should?) be very different from your case number 1.
--Michael
> On 11.06.2014, at 23:13, Prakash Gowri Shankor <pr...@gmail.com> wrote:
>
> Thanks for your response Michael.
>
> In step 3, I am actually stopping the entire cluster and restarting it
> without the 2nd broker. But I see your point. When i look in
> /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it
> holds test2-1 ( ie 1st partition of test2 topic ).
> For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it
> holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ).
> So it would seem that kafka is missing the leader for partition 1 and hence
> throwing the exception on the producer side.
> Let me try your replication suggestion.
>
> While all of the above might explain the exception in the case of 2
> brokers, there are still times when I see it with just a single broker.
> In this case, I start from a normal working cluster with 1 broker only.
> Then I either put my machine into sleep/hibernation. On wake, I do shutdown
> the cluster ( for sanity ) and restart.
> On restart, I start seeing this exception. In this case i only have one
> broker. I still create the topic the way i described earlier.
> I understand this is not the ideal production topology, but its annoying to
> see it during development.
>
> Thanks
>
>
> On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll <mi...@michael-noll.com>
> wrote:
>
>> Prakash,
>>
>> you are configure the topic with a replication factor of only 1, i.e. no
>> additional replica beyond "the original one". This replication setting
>> of 1 means that only one of the two brokers will ever host the (single)
>> replica -- which is implied to also be the leader in-sync replica -- of
>> a given partition.
>>
>> In step 3 you are disabling one of the two brokers. Because this
>> stopped broker is the only broker that hosts one or more of the 3
>> partitions you configured (I can't tell which partition(s) it is, but
>> you can find out by --describe'ing the topic), your Kafka cluster --
>> which is now running in degraded state -- will miss the leader of those
>> affected partitions. And because you set the replication factor to 1,
>> the remaining, second broker will not and will never take over the
>> leadership of those partitions from the stopped broker. Hence you will
>> keep getting the LeaderNotAvailableException's until you restart the
>> stopped broker in step 7.
>>
>> So to me it looks as if the behavior of Kafka is actually correct and as
>> expected.
>>
>> If you want to "rectify" your test setup, try increasing the replication
>> factor from 1 to 2. If you do, you should be able to go through steps
>> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
>> Kafka some time to re-elect the remaining, second broker as the new
>> leader for the first broker's partitions though).
>>
>> Hope this helps,
>> Michael
>>
>>
>>
>>> On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
>>> yes,
>>> here are the steps:
>>>
>>> Create topic as : ./kafka-topics.sh --topic test2 --create
>> --partitions 3
>>> --zookeeper localhost:2181 --replication-factor 1
>>>
>>> 1) Start cluster with 2 brokers, 3 consumers.
>>> 2) Dont start any producer
>>> 3) Shutdown cluster and disable one broker from starting
>>> 4) restart cluster with 1 broker, 3 consumers
>>> 5) Start producer and send messages. I see this exception
>>> 6) Shutdown cluster.
>>> 7) Enable 2nd broker.
>>> 8) Restart cluster with 2 brokers, 3 consumer and the one producer and
>> send
>>> messages. Now I dont see the exception.
>>
>>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
Thanks for your response Michael.
In step 3, I am actually stopping the entire cluster and restarting it
without the 2nd broker. But I see your point. When i look in
/tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it
holds test2-1 ( ie 1st partition of test2 topic ).
For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it
holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ).
So it would seem that kafka is missing the leader for partition 1 and hence
throwing the exception on the producer side.
Let me try your replication suggestion.
While all of the above might explain the exception in the case of 2
brokers, there are still times when I see it with just a single broker.
In this case, I start from a normal working cluster with 1 broker only.
Then I either put my machine into sleep/hibernation. On wake, I do shutdown
the cluster ( for sanity ) and restart.
On restart, I start seeing this exception. In this case i only have one
broker. I still create the topic the way i described earlier.
I understand this is not the ideal production topology, but its annoying to
see it during development.
Thanks
On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll <mi...@michael-noll.com>
wrote:
> Prakash,
>
> you are configure the topic with a replication factor of only 1, i.e. no
> additional replica beyond "the original one". This replication setting
> of 1 means that only one of the two brokers will ever host the (single)
> replica -- which is implied to also be the leader in-sync replica -- of
> a given partition.
>
> In step 3 you are disabling one of the two brokers. Because this
> stopped broker is the only broker that hosts one or more of the 3
> partitions you configured (I can't tell which partition(s) it is, but
> you can find out by --describe'ing the topic), your Kafka cluster --
> which is now running in degraded state -- will miss the leader of those
> affected partitions. And because you set the replication factor to 1,
> the remaining, second broker will not and will never take over the
> leadership of those partitions from the stopped broker. Hence you will
> keep getting the LeaderNotAvailableException's until you restart the
> stopped broker in step 7.
>
> So to me it looks as if the behavior of Kafka is actually correct and as
> expected.
>
> If you want to "rectify" your test setup, try increasing the replication
> factor from 1 to 2. If you do, you should be able to go through steps
> 1-8 without seeing LeaderNotAvailableExceptions (you may need to give
> Kafka some time to re-elect the remaining, second broker as the new
> leader for the first broker's partitions though).
>
> Hope this helps,
> Michael
>
>
>
> On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
> > yes,
> > here are the steps:
> >
> > Create topic as : ./kafka-topics.sh --topic test2 --create
> --partitions 3
> > --zookeeper localhost:2181 --replication-factor 1
> >
> > 1) Start cluster with 2 brokers, 3 consumers.
> > 2) Dont start any producer
> > 3) Shutdown cluster and disable one broker from starting
> > 4) restart cluster with 1 broker, 3 consumers
> > 5) Start producer and send messages. I see this exception
> > 6) Shutdown cluster.
> > 7) Enable 2nd broker.
> > 8) Restart cluster with 2 brokers, 3 consumer and the one producer and
> send
> > messages. Now I dont see the exception.
>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
Prakash,
you are configure the topic with a replication factor of only 1, i.e. no
additional replica beyond "the original one". This replication setting
of 1 means that only one of the two brokers will ever host the (single)
replica -- which is implied to also be the leader in-sync replica -- of
a given partition.
In step 3 you are disabling one of the two brokers. Because this
stopped broker is the only broker that hosts one or more of the 3
partitions you configured (I can't tell which partition(s) it is, but
you can find out by --describe'ing the topic), your Kafka cluster --
which is now running in degraded state -- will miss the leader of those
affected partitions. And because you set the replication factor to 1,
the remaining, second broker will not and will never take over the
leadership of those partitions from the stopped broker. Hence you will
keep getting the LeaderNotAvailableException's until you restart the
stopped broker in step 7.
So to me it looks as if the behavior of Kafka is actually correct and as
expected.
If you want to "rectify" your test setup, try increasing the replication
factor from 1 to 2. If you do, you should be able to go through steps
1-8 without seeing LeaderNotAvailableExceptions (you may need to give
Kafka some time to re-elect the remaining, second broker as the new
leader for the first broker's partitions though).
Hope this helps,
Michael
On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote:
> yes,
> here are the steps:
>
> Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3
> --zookeeper localhost:2181 --replication-factor 1
>
> 1) Start cluster with 2 brokers, 3 consumers.
> 2) Dont start any producer
> 3) Shutdown cluster and disable one broker from starting
> 4) restart cluster with 1 broker, 3 consumers
> 5) Start producer and send messages. I see this exception
> 6) Shutdown cluster.
> 7) Enable 2nd broker.
> 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send
> messages. Now I dont see the exception.
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
yes,
here are the steps:
Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3
--zookeeper localhost:2181 --replication-factor 1
1) Start cluster with 2 brokers, 3 consumers.
2) Dont start any producer
3) Shutdown cluster and disable one broker from starting
4) restart cluster with 1 broker, 3 consumers
5) Start producer and send messages. I see this exception
6) Shutdown cluster.
7) Enable 2nd broker.
8) Restart cluster with 2 brokers, 3 consumer and the one producer and send
messages. Now I dont see the exception.
On Tue, Jun 10, 2014 at 2:56 PM, Joel Koshy <jj...@gmail.com> wrote:
> Do you by any chance have steps to reproduce this issue easily?
>
> On Tue, Jun 10, 2014 at 02:23:20PM -0700, Prakash Gowri Shankor wrote:
> > No i have not used the delete topic feature. I have been manually
> deleting
> > the topics from zookeeper and removing the topic from the kafka and zk
> logs.
> > I've experimented a bit more. It seems like this occurs when I have a
> > single broker running. When i restart with 2 brokers, the problem goes
> away.
> >
> >
> > On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> > > Did you use the delete topic command?
> > >
> > > That was an experimental feature in the 0.8.1 release with several
> > > bugs. The fixes are all on trunk, but those fixes did not make it into
> > > 0.8.1.1 - except for a config option to disable delete-topic support
> > > on the broker.
> > >
> > > Joel
> > >
> > > On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
> > > > From the moment it starts occuring, it is persistent. Restarts dont
> seem
> > > to
> > > > make it go away. The only thing that makes it go away is following
> the
> > > > steps listed in this stackoverflow thread.
> > > >
> > > >
> > >
> http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
> > > >
> > > >
> > > >
> > > > On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Prakash,
> > > > >
> > > > > Is this exception transient or persistent on broker startup?
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> > > > > prakash.shankor@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am running a cluster with a single broker, the performance
> producer
> > > > > > script and 3 consumers.
> > > > > > On a fresh start of the cluster , the producer throws this
> exception.
> > > > > > I was able to run this cluster successfully on the same topic (
> > > test2 )
> > > > > > successfully the first time.
> > > > > >
> > > > > > The solution( from stackoverflow ) seems to be to delete the
> topic
> > > data
> > > > > in
> > > > > > the broker and in zookeeper. This doesnt seem to be a viable
> > > production
> > > > > > solution to me. Is there a way to solve this without losing topic
> > > data ?
> > > > > >
> > > > > > Also does the incidence of this problem decrease if I run more
> > > > > > brokers/servers ?
> > > > > >
> > > > > > I see logs like these in the server.log
> > > > > >
> > > > > > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> > > > > correlation
> > > > > > id 0 from client on partition [test2,0] failed due to Topic
> test2
> > > either
> > > > > > doesn't exist or is in the process of being deleted
> > > > > > (kafka.server.KafkaApis)
> > > > > >
> > > > > > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> > > > > correlation
> > > > > > id 0 from client on partition [test2,1] failed due to Topic
> test2
> > > either
> > > > > > doesn't exist or is in the process of being deleted
> > > > > > (kafka.server.KafkaApis)
> > > > > >
> > > > > > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> > > > > correlation
> > > > > > id 0 from client on partition [test2,2] failed due to Topic
> test2
> > > either
> > > > > > doesn't exist or is in the process of being deleted
> > > > > > (kafka.server.KafkaApis)
> > > > > >
> > > > > > The exception trace is:
> > > > > >
> > > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
> events
> > > > > > (kafka.producer.async.ProducerSendThread)
> > > > > >
> > > > > > kafka.common.FailedToSendMessageException: Failed to send
> messages
> > > after
> > > > > 3
> > > > > > tries.
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > > >
> > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > > >
> > > > > > at
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > > >
> > > > > > [2014-06-10 10:45:32,464] ERROR Failed to send requests for
> topics
> > > test2
> > > > > > with correlation ids in [41,48]
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > >
> > > > > > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test2 ->
> > > > > >
> > > > > > No partition metadata for topic test2 due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test2]:
> class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > >
> > > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
> events
> > > > > > (kafka.producer.async.ProducerSendThread)
> > > > > >
> > > > > > kafka.common.FailedToSendMessageException: Failed to send
> messages
> > > after
> > > > > 3
> > > > > > tries.
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > > >
> > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > > >
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > > >
> > > > > > at
> > > > >
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > >
> > >
>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Joel Koshy <jj...@gmail.com>.
Do you by any chance have steps to reproduce this issue easily?
On Tue, Jun 10, 2014 at 02:23:20PM -0700, Prakash Gowri Shankor wrote:
> No i have not used the delete topic feature. I have been manually deleting
> the topics from zookeeper and removing the topic from the kafka and zk logs.
> I've experimented a bit more. It seems like this occurs when I have a
> single broker running. When i restart with 2 brokers, the problem goes away.
>
>
> On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > Did you use the delete topic command?
> >
> > That was an experimental feature in the 0.8.1 release with several
> > bugs. The fixes are all on trunk, but those fixes did not make it into
> > 0.8.1.1 - except for a config option to disable delete-topic support
> > on the broker.
> >
> > Joel
> >
> > On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
> > > From the moment it starts occuring, it is persistent. Restarts dont seem
> > to
> > > make it go away. The only thing that makes it go away is following the
> > > steps listed in this stackoverflow thread.
> > >
> > >
> > http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
> > >
> > >
> > >
> > > On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hello Prakash,
> > > >
> > > > Is this exception transient or persistent on broker startup?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> > > > prakash.shankor@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am running a cluster with a single broker, the performance producer
> > > > > script and 3 consumers.
> > > > > On a fresh start of the cluster , the producer throws this exception.
> > > > > I was able to run this cluster successfully on the same topic (
> > test2 )
> > > > > successfully the first time.
> > > > >
> > > > > The solution( from stackoverflow ) seems to be to delete the topic
> > data
> > > > in
> > > > > the broker and in zookeeper. This doesnt seem to be a viable
> > production
> > > > > solution to me. Is there a way to solve this without losing topic
> > data ?
> > > > >
> > > > > Also does the incidence of this problem decrease if I run more
> > > > > brokers/servers ?
> > > > >
> > > > > I see logs like these in the server.log
> > > > >
> > > > > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,0] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,1] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> > > > correlation
> > > > > id 0 from client on partition [test2,2] failed due to Topic test2
> > either
> > > > > doesn't exist or is in the process of being deleted
> > > > > (kafka.server.KafkaApis)
> > > > >
> > > > > The exception trace is:
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > > > (kafka.producer.async.ProducerSendThread)
> > > > >
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after
> > > > 3
> > > > > tries.
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > >
> > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > >
> > > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
> > test2
> > > > > with correlation ids in [41,48]
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > >
> > > > > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test2 ->
> > > > >
> > > > > No partition metadata for topic test2 due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > >
> > > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > > > (kafka.producer.async.ProducerSendThread)
> > > > >
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after
> > > > 3
> > > > > tries.
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > >
> > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > >
> > > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> >
> >
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
No i have not used the delete topic feature. I have been manually deleting
the topics from zookeeper and removing the topic from the kafka and zk logs.
I've experimented a bit more. It seems like this occurs when I have a
single broker running. When i restart with 2 brokers, the problem goes away.
On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy <jj...@gmail.com> wrote:
> Did you use the delete topic command?
>
> That was an experimental feature in the 0.8.1 release with several
> bugs. The fixes are all on trunk, but those fixes did not make it into
> 0.8.1.1 - except for a config option to disable delete-topic support
> on the broker.
>
> Joel
>
> On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
> > From the moment it starts occuring, it is persistent. Restarts dont seem
> to
> > make it go away. The only thing that makes it go away is following the
> > steps listed in this stackoverflow thread.
> >
> >
> http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
> >
> >
> >
> > On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hello Prakash,
> > >
> > > Is this exception transient or persistent on broker startup?
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> > > prakash.shankor@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am running a cluster with a single broker, the performance producer
> > > > script and 3 consumers.
> > > > On a fresh start of the cluster , the producer throws this exception.
> > > > I was able to run this cluster successfully on the same topic (
> test2 )
> > > > successfully the first time.
> > > >
> > > > The solution( from stackoverflow ) seems to be to delete the topic
> data
> > > in
> > > > the broker and in zookeeper. This doesnt seem to be a viable
> production
> > > > solution to me. Is there a way to solve this without losing topic
> data ?
> > > >
> > > > Also does the incidence of this problem decrease if I run more
> > > > brokers/servers ?
> > > >
> > > > I see logs like these in the server.log
> > > >
> > > > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> > > correlation
> > > > id 0 from client on partition [test2,0] failed due to Topic test2
> either
> > > > doesn't exist or is in the process of being deleted
> > > > (kafka.server.KafkaApis)
> > > >
> > > > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> > > correlation
> > > > id 0 from client on partition [test2,1] failed due to Topic test2
> either
> > > > doesn't exist or is in the process of being deleted
> > > > (kafka.server.KafkaApis)
> > > >
> > > > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> > > correlation
> > > > id 0 from client on partition [test2,2] failed due to Topic test2
> either
> > > > doesn't exist or is in the process of being deleted
> > > > (kafka.server.KafkaApis)
> > > >
> > > > The exception trace is:
> > > >
> > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > > (kafka.producer.async.ProducerSendThread)
> > > >
> > > > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> > > 3
> > > > tries.
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > >
> > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > >
> > > > at
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > >
> > > > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
> test2
> > > > with correlation ids in [41,48]
> > > (kafka.producer.async.DefaultEventHandler)
> > > >
> > > > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test2 ->
> > > >
> > > > No partition metadata for topic test2 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > >
> > > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > > (kafka.producer.async.ProducerSendThread)
> > > >
> > > > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> > > 3
> > > > tries.
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > >
> > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > > >
> > > > at
> > > >
> > > >
> > >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > >
> > > > at
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
>
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Joel Koshy <jj...@gmail.com>.
Did you use the delete topic command?
That was an experimental feature in the 0.8.1 release with several
bugs. The fixes are all on trunk, but those fixes did not make it into
0.8.1.1 - except for a config option to disable delete-topic support
on the broker.
Joel
On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
> From the moment it starts occuring, it is persistent. Restarts dont seem to
> make it go away. The only thing that makes it go away is following the
> steps listed in this stackoverflow thread.
>
> http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
>
>
>
> On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hello Prakash,
> >
> > Is this exception transient or persistent on broker startup?
> >
> > Guozhang
> >
> >
> > On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> > prakash.shankor@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am running a cluster with a single broker, the performance producer
> > > script and 3 consumers.
> > > On a fresh start of the cluster , the producer throws this exception.
> > > I was able to run this cluster successfully on the same topic ( test2 )
> > > successfully the first time.
> > >
> > > The solution( from stackoverflow ) seems to be to delete the topic data
> > in
> > > the broker and in zookeeper. This doesnt seem to be a viable production
> > > solution to me. Is there a way to solve this without losing topic data ?
> > >
> > > Also does the incidence of this problem decrease if I run more
> > > brokers/servers ?
> > >
> > > I see logs like these in the server.log
> > >
> > > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> > correlation
> > > id 0 from client on partition [test2,0] failed due to Topic test2 either
> > > doesn't exist or is in the process of being deleted
> > > (kafka.server.KafkaApis)
> > >
> > > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> > correlation
> > > id 0 from client on partition [test2,1] failed due to Topic test2 either
> > > doesn't exist or is in the process of being deleted
> > > (kafka.server.KafkaApis)
> > >
> > > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> > correlation
> > > id 0 from client on partition [test2,2] failed due to Topic test2 either
> > > doesn't exist or is in the process of being deleted
> > > (kafka.server.KafkaApis)
> > >
> > > The exception trace is:
> > >
> > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > (kafka.producer.async.ProducerSendThread)
> > >
> > > kafka.common.FailedToSendMessageException: Failed to send messages after
> > 3
> > > tries.
> > >
> > > at
> > >
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > >
> > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > >
> > > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > >
> > > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
> > > with correlation ids in [41,48]
> > (kafka.producer.async.DefaultEventHandler)
> > >
> > > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test2 ->
> > >
> > > No partition metadata for topic test2 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > >
> > > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > > (kafka.producer.async.ProducerSendThread)
> > >
> > > kafka.common.FailedToSendMessageException: Failed to send messages after
> > 3
> > > tries.
> > >
> > > at
> > >
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > >
> > > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> > >
> > > at
> > >
> > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > >
> > > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Prakash Gowri Shankor <pr...@gmail.com>.
>From the moment it starts occuring, it is persistent. Restarts dont seem to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Prakash,
>
> Is this exception transient or persistent on broker startup?
>
> Guozhang
>
>
> On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
> prakash.shankor@gmail.com> wrote:
>
> > Hi,
> >
> > I am running a cluster with a single broker, the performance producer
> > script and 3 consumers.
> > On a fresh start of the cluster , the producer throws this exception.
> > I was able to run this cluster successfully on the same topic ( test2 )
> > successfully the first time.
> >
> > The solution( from stackoverflow ) seems to be to delete the topic data
> in
> > the broker and in zookeeper. This doesnt seem to be a viable production
> > solution to me. Is there a way to solve this without losing topic data ?
> >
> > Also does the incidence of this problem decrease if I run more
> > brokers/servers ?
> >
> > I see logs like these in the server.log
> >
> > [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,0] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,1] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
> correlation
> > id 0 from client on partition [test2,2] failed due to Topic test2 either
> > doesn't exist or is in the process of being deleted
> > (kafka.server.KafkaApis)
> >
> > The exception trace is:
> >
> > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > (kafka.producer.async.ProducerSendThread)
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> >
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> >
> > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> >
> > at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> > [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
> > with correlation ids in [41,48]
> (kafka.producer.async.DefaultEventHandler)
> >
> > [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> > [{TopicMetadata for topic test2 ->
> >
> > No partition metadata for topic test2 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> >
> > [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> > (kafka.producer.async.ProducerSendThread)
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> >
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> >
> > at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> >
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> >
> > at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
>
>
>
> --
> -- Guozhang
>
Re: LeaderNotAvailableException in 0.8.1.1
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor <
prakash.shankor@gmail.com> wrote:
> Hi,
>
> I am running a cluster with a single broker, the performance producer
> script and 3 consumers.
> On a fresh start of the cluster , the producer throws this exception.
> I was able to run this cluster successfully on the same topic ( test2 )
> successfully the first time.
>
> The solution( from stackoverflow ) seems to be to delete the topic data in
> the broker and in zookeeper. This doesnt seem to be a viable production
> solution to me. Is there a way to solve this without losing topic data ?
>
> Also does the incidence of this problem decrease if I run more
> brokers/servers ?
>
> I see logs like these in the server.log
>
> [2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,0] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> [2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,1] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> [2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with correlation
> id 0 from client on partition [test2,2] failed due to Topic test2 either
> doesn't exist or is in the process of being deleted
> (kafka.server.KafkaApis)
>
> The exception trace is:
>
> [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> (kafka.producer.async.ProducerSendThread)
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
>
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
>
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
> [2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
> with correlation ids in [41,48] (kafka.producer.async.DefaultEventHandler)
>
> [2014-06-10 10:45:32,464] WARN Error while fetching metadata
> [{TopicMetadata for topic test2 ->
>
> No partition metadata for topic test2 due to
> kafka.common.LeaderNotAvailableException}] for topic [test2]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
>
> [2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
> (kafka.producer.async.ProducerSendThread)
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
>
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
>
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
--
-- Guozhang