You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Arjun <ar...@socialtwist.com> on 2014/02/21 11:33:34 UTC
producer not pushing the data into broker
hi,
I am testing kafka 0.8 on my local machine. i have only one zookeeper
and one kafka broker running.
when i run the console producer i get this error:
[2014-02-21 16:01:20,512] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-02-21 16:01:20,632] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3 tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
Can some one please let me know why this is happening.
I checked my vhost files and localhost, my hostname are correct.
Re: producer not pushing the data into broker
Posted by Arjun <ar...@socialtwist.com>.
The consumer offset checker gives this error
arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
localhost:2181,localhost:2182,localhost:2183 --topic test
Group Topic Pid Offset
logSize Lag Owner
Exception in thread "main"
org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /consumers/group1/offsets/test/0
at
org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
at kafka.utils.ZkUtils$.readData(ZkUtils.scala:459)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:89)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:88)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at
kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:152)
at kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/group1/offsets/test/0
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 16 more
On Saturday 22 February 2014 02:24 PM, Arjun wrote:
> Hi please find below the o/p of the list and the console producer.
> There are no errors in the state change log or in the controller log.
>
> One thing i found is this one happens only at starting time. I mean
> the first time, when i start kafka and try to insert the messages i
> see this error.
>
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> group1 --zkconnect localhost:2181,localhost:2182,localhost:2183
> --topic test
> Group Topic Pid Offset
> logSize Lag Owner
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
> bin/kafka-console-producer.sh --broker-list
> localhost:9092,localhost:9093,localhost:9094 --topic test
> THis ia s test message
> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics
> test with correlation ids in [0,8]
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages
> after 3 tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
> bin/kafka-list-topic.sh --zookeeper localhost:2181
> topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
> topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
> topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
> topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
> topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
> topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
> topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
> topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
> topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
> topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
> topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
> topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
>
>
> What i did:
> started 3 kafka processes in my system.The config is default config
> with just the ports and log dir's changed.There are 3 zookeeper
> servers running.
> If i start the console producer next time it will work fine.
> To reproduce what did was, stopped all the kafka and zookeeper
> process, deleted the log dirs of the kafka and the zookeeper and
> restated the processes.
>
>
> Thanks
> Arjun Narasimha Kota
>
>
> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>> Could you do a list topic and show the output? Also, any error in the
>> controller and state-change log?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>>
>>> hi,
>>>
>>> I am testing kafka 0.8 on my local machine. i have only one
>>> zookeeper and
>>> one kafka broker running.
>>>
>>> when i run the console producer i get this error:
>>>
>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>> No partition metadata for topic test due to
>>> kafka.common.LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>> No partition metadata for topic test due to
>>> kafka.common.LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>>> with correlation ids in [0,8]
>>> (kafka.producer.async.DefaultEventHandler)
>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>> ProducerSendThread)
>>> kafka.common.FailedToSendMessageException: Failed to send messages
>>> after
>>> 3 tries.
>>> at kafka.producer.async.DefaultEventHandler.handle(
>>> DefaultEventHandler.scala:90)
>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>> ProducerSendThread.scala:104)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>> ProducerSendThread.scala:66)
>>> at kafka.producer.async.ProducerSendThread.run(
>>> ProducerSendThread.scala:44)
>>>
>>> Can some one please let me know why this is happening.
>>> I checked my vhost files and localhost, my hostname are correct.
>>>
>
Re: producer not pushing the data into broker
Posted by Arjun <ar...@socialtwist.com>.
Hi,
thanks for the input. I will try to increase the number of retries and
check.
Thanks
Arjun Narasimha kota
On Monday 24 February 2014 10:44 AM, Jun Rao wrote:
> If those errors only show up transiently when brokers are started, then
> it's normal. It takes a bit of time for metadata to be probably to the
> newly started brokers. You may have to increase # retries to avoid the
> failure in the producer.
>
> Thanks,
>
> Jun
>
>
> On Sun, Feb 23, 2014 at 6:33 PM, Arjun <ar...@socialtwist.com> wrote:
>
>> Hi,
>> No i have 3 kafka brokers running, but in the same system. I tried with
>> the replication factor of 1 but it gives the same result.
>>
>> Thanks
>> Arjun Narasimha Kota
>>
>>
>> On Monday 24 February 2014 04:21 AM, Jun Rao wrote:
>>
>>> With only one broker, do you really want to have replication factor 2?
>>> Maybe you could retry with replication factor 1.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>>
>>>
>>> On Sat, Feb 22, 2014 at 12:54 AM, Arjun <ar...@socialtwist.com> wrote:
>>>
>>> Hi please find below the o/p of the list and the console producer. There
>>>> are no errors in the state change log or in the controller log.
>>>>
>>>> One thing i found is this one happens only at starting time. I mean the
>>>> first time, when i start kafka and try to insert the messages i see this
>>>> error.
>>>>
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
>>>> kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
>>>> localhost:2181,localhost:2182,localhost:2183 --topic test
>>>> Group Topic Pid Offset logSize
>>>> Lag Owner
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>>> bin/kafka-console-producer.sh --broker-list
>>>> localhost:9092,localhost:9093,localhost:9094
>>>> --topic test
>>>> THis ia s test message
>>>> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>>
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
>>>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
>>>> (kafka.producer.async.ProducerSendThread)
>>>>
>>>> kafka.common.FailedToSendMessageException: Failed to send messages after
>>>> 3 tries.
>>>> at kafka.producer.async.DefaultEventHandler.handle(
>>>> DefaultEventHandler.scala:90)
>>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>> ProducerSendThread.scala:104)
>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>>> ProducerSendThread.scala:66)
>>>> at kafka.producer.async.ProducerSendThread.run(
>>>> ProducerSendThread.scala:44)
>>>>
>>>>
>>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>>> bin/kafka-list-topic.sh
>>>> --zookeeper localhost:2181
>>>> topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
>>>> topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
>>>> topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
>>>> topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
>>>> topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
>>>> topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
>>>> topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
>>>> topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
>>>> topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
>>>> topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
>>>> topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
>>>> topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
>>>>
>>>>
>>>> What i did:
>>>> started 3 kafka processes in my system.The config is default config with
>>>> just the ports and log dir's changed.There are 3 zookeeper servers
>>>> running.
>>>> If i start the console producer next time it will work fine.
>>>> To reproduce what did was, stopped all the kafka and zookeeper process,
>>>> deleted the log dirs of the kafka and the zookeeper and restated the
>>>> processes.
>>>>
>>>>
>>>> Thanks
>>>> Arjun Narasimha Kota
>>>>
>>>>
>>>>
>>>> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>>>>
>>>> Could you do a list topic and show the output? Also, any error in the
>>>>> controller and state-change log?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>>>>>
>>>>> hi,
>>>>>
>>>>>> I am testing kafka 0.8 on my local machine. i have only one zookeeper
>>>>>> and
>>>>>> one kafka broker running.
>>>>>>
>>>>>> when i run the console producer i get this error:
>>>>>>
>>>>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>>>>> [{TopicMetadata for topic test ->
>>>>>> No partition metadata for topic test due to kafka.common.
>>>>>> LeaderNotAvailableException}]
>>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>>>> (kafka.producer.async.DefaultEventHandler)
>>>>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>>>>> [{TopicMetadata for topic test ->
>>>>>> No partition metadata for topic test due to kafka.common.
>>>>>> LeaderNotAvailableException}]
>>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>>>>>> with correlation ids in [0,8] (kafka.producer.async.
>>>>>> DefaultEventHandler)
>>>>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>>>>> ProducerSendThread)
>>>>>> kafka.common.FailedToSendMessageException: Failed to send messages
>>>>>> after
>>>>>> 3 tries.
>>>>>> at kafka.producer.async.DefaultEventHandler.handle(
>>>>>> DefaultEventHandler.scala:90)
>>>>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>>>> ProducerSendThread.scala:104)
>>>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>>>>> ProducerSendThread.scala:66)
>>>>>> at kafka.producer.async.ProducerSendThread.run(
>>>>>> ProducerSendThread.scala:44)
>>>>>>
>>>>>> Can some one please let me know why this is happening.
>>>>>> I checked my vhost files and localhost, my hostname are correct.
>>>>>>
>>>>>>
>>>>>>
Re: producer not pushing the data into broker
Posted by Jun Rao <ju...@gmail.com>.
If those errors only show up transiently when brokers are started, then
it's normal. It takes a bit of time for metadata to be probably to the
newly started brokers. You may have to increase # retries to avoid the
failure in the producer.
Thanks,
Jun
On Sun, Feb 23, 2014 at 6:33 PM, Arjun <ar...@socialtwist.com> wrote:
> Hi,
> No i have 3 kafka brokers running, but in the same system. I tried with
> the replication factor of 1 but it gives the same result.
>
> Thanks
> Arjun Narasimha Kota
>
>
> On Monday 24 February 2014 04:21 AM, Jun Rao wrote:
>
>> With only one broker, do you really want to have replication factor 2?
>> Maybe you could retry with replication factor 1.
>>
>> Thanks,
>>
>> Jun
>>
>>
>>
>>
>> On Sat, Feb 22, 2014 at 12:54 AM, Arjun <ar...@socialtwist.com> wrote:
>>
>> Hi please find below the o/p of the list and the console producer. There
>>> are no errors in the state change log or in the controller log.
>>>
>>> One thing i found is this one happens only at starting time. I mean the
>>> first time, when i start kafka and try to insert the messages i see this
>>> error.
>>>
>>>
>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
>>> kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
>>> localhost:2181,localhost:2182,localhost:2183 --topic test
>>> Group Topic Pid Offset logSize
>>> Lag Owner
>>>
>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>> bin/kafka-console-producer.sh --broker-list
>>> localhost:9092,localhost:9093,localhost:9094
>>> --topic test
>>> THis ia s test message
>>> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>>
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
>>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
>>> (kafka.producer.async.ProducerSendThread)
>>>
>>> kafka.common.FailedToSendMessageException: Failed to send messages after
>>> 3 tries.
>>> at kafka.producer.async.DefaultEventHandler.handle(
>>> DefaultEventHandler.scala:90)
>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>> ProducerSendThread.scala:104)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>> ProducerSendThread.scala:66)
>>> at kafka.producer.async.ProducerSendThread.run(
>>> ProducerSendThread.scala:44)
>>>
>>>
>>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>>> bin/kafka-list-topic.sh
>>> --zookeeper localhost:2181
>>> topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
>>> topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
>>> topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
>>> topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
>>> topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
>>> topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
>>> topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
>>> topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
>>> topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
>>> topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
>>> topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
>>> topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
>>>
>>>
>>> What i did:
>>> started 3 kafka processes in my system.The config is default config with
>>> just the ports and log dir's changed.There are 3 zookeeper servers
>>> running.
>>> If i start the console producer next time it will work fine.
>>> To reproduce what did was, stopped all the kafka and zookeeper process,
>>> deleted the log dirs of the kafka and the zookeeper and restated the
>>> processes.
>>>
>>>
>>> Thanks
>>> Arjun Narasimha Kota
>>>
>>>
>>>
>>> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>>>
>>> Could you do a list topic and show the output? Also, any error in the
>>>> controller and state-change log?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>>>>
>>>> hi,
>>>>
>>>>> I am testing kafka 0.8 on my local machine. i have only one zookeeper
>>>>> and
>>>>> one kafka broker running.
>>>>>
>>>>> when i run the console producer i get this error:
>>>>>
>>>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>>>> [{TopicMetadata for topic test ->
>>>>> No partition metadata for topic test due to kafka.common.
>>>>> LeaderNotAvailableException}]
>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>>> (kafka.producer.async.DefaultEventHandler)
>>>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>>>> [{TopicMetadata for topic test ->
>>>>> No partition metadata for topic test due to kafka.common.
>>>>> LeaderNotAvailableException}]
>>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>>> (kafka.producer.BrokerPartitionInfo)
>>>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>>>>> with correlation ids in [0,8] (kafka.producer.async.
>>>>> DefaultEventHandler)
>>>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>>>> ProducerSendThread)
>>>>> kafka.common.FailedToSendMessageException: Failed to send messages
>>>>> after
>>>>> 3 tries.
>>>>> at kafka.producer.async.DefaultEventHandler.handle(
>>>>> DefaultEventHandler.scala:90)
>>>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>>> ProducerSendThread.scala:104)
>>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>>>> ProducerSendThread.scala:66)
>>>>> at kafka.producer.async.ProducerSendThread.run(
>>>>> ProducerSendThread.scala:44)
>>>>>
>>>>> Can some one please let me know why this is happening.
>>>>> I checked my vhost files and localhost, my hostname are correct.
>>>>>
>>>>>
>>>>>
>
Re: producer not pushing the data into broker
Posted by Arjun <ar...@socialtwist.com>.
Hi,
No i have 3 kafka brokers running, but in the same system. I tried with
the replication factor of 1 but it gives the same result.
Thanks
Arjun Narasimha Kota
On Monday 24 February 2014 04:21 AM, Jun Rao wrote:
> With only one broker, do you really want to have replication factor 2?
> Maybe you could retry with replication factor 1.
>
> Thanks,
>
> Jun
>
>
>
>
> On Sat, Feb 22, 2014 at 12:54 AM, Arjun <ar...@socialtwist.com> wrote:
>
>> Hi please find below the o/p of the list and the console producer. There
>> are no errors in the state change log or in the controller log.
>>
>> One thing i found is this one happens only at starting time. I mean the
>> first time, when i start kafka and try to insert the messages i see this
>> error.
>>
>>
>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
>> kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
>> localhost:2181,localhost:2182,localhost:2183 --topic test
>> Group Topic Pid Offset logSize
>> Lag Owner
>>
>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
>> bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094
>> --topic test
>> THis ia s test message
>> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
>> partition due to: Failed to fetch topic metadata for topic: test
>> (kafka.producer.async.DefaultEventHandler)
>> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
>> partition due to: Failed to fetch topic metadata for topic: test
>> (kafka.producer.async.DefaultEventHandler)
>> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
>> partition due to: Failed to fetch topic metadata for topic: test
>> (kafka.producer.async.DefaultEventHandler)
>> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
>> partition due to: Failed to fetch topic metadata for topic: test
>> (kafka.producer.async.DefaultEventHandler)
>> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>>
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
>> (kafka.producer.async.ProducerSendThread)
>>
>> kafka.common.FailedToSendMessageException: Failed to send messages after
>> 3 tries.
>> at kafka.producer.async.DefaultEventHandler.handle(
>> DefaultEventHandler.scala:90)
>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>> ProducerSendThread.scala:104)
>> at kafka.producer.async.ProducerSendThread$$anonfun$
>> processEvents$3.apply(ProducerSendThread.scala:87)
>> at kafka.producer.async.ProducerSendThread$$anonfun$
>> processEvents$3.apply(ProducerSendThread.scala:67)
>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>> at kafka.producer.async.ProducerSendThread.processEvents(
>> ProducerSendThread.scala:66)
>> at kafka.producer.async.ProducerSendThread.run(
>> ProducerSendThread.scala:44)
>>
>>
>> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-list-topic.sh
>> --zookeeper localhost:2181
>> topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
>> topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
>> topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
>> topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
>> topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
>> topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
>> topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
>> topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
>> topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
>> topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
>> topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
>> topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
>>
>>
>> What i did:
>> started 3 kafka processes in my system.The config is default config with
>> just the ports and log dir's changed.There are 3 zookeeper servers running.
>> If i start the console producer next time it will work fine.
>> To reproduce what did was, stopped all the kafka and zookeeper process,
>> deleted the log dirs of the kafka and the zookeeper and restated the
>> processes.
>>
>>
>> Thanks
>> Arjun Narasimha Kota
>>
>>
>>
>> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>>
>>> Could you do a list topic and show the output? Also, any error in the
>>> controller and state-change log?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>>>
>>> hi,
>>>> I am testing kafka 0.8 on my local machine. i have only one zookeeper and
>>>> one kafka broker running.
>>>>
>>>> when i run the console producer i get this error:
>>>>
>>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>>> partition due to: Failed to fetch topic metadata for topic: test
>>>> (kafka.producer.async.DefaultEventHandler)
>>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>>> [{TopicMetadata for topic test ->
>>>> No partition metadata for topic test due to kafka.common.
>>>> LeaderNotAvailableException}]
>>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>>> (kafka.producer.BrokerPartitionInfo)
>>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>>>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>>> ProducerSendThread)
>>>> kafka.common.FailedToSendMessageException: Failed to send messages after
>>>> 3 tries.
>>>> at kafka.producer.async.DefaultEventHandler.handle(
>>>> DefaultEventHandler.scala:90)
>>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>>> ProducerSendThread.scala:104)
>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>>> ProducerSendThread.scala:66)
>>>> at kafka.producer.async.ProducerSendThread.run(
>>>> ProducerSendThread.scala:44)
>>>>
>>>> Can some one please let me know why this is happening.
>>>> I checked my vhost files and localhost, my hostname are correct.
>>>>
>>>>
Re: producer not pushing the data into broker
Posted by Jun Rao <ju...@gmail.com>.
With only one broker, do you really want to have replication factor 2?
Maybe you could retry with replication factor 1.
Thanks,
Jun
On Sat, Feb 22, 2014 at 12:54 AM, Arjun <ar...@socialtwist.com> wrote:
> Hi please find below the o/p of the list and the console producer. There
> are no errors in the state change log or in the controller log.
>
> One thing i found is this one happens only at starting time. I mean the
> first time, when i start kafka and try to insert the messages i see this
> error.
>
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
> kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
> localhost:2181,localhost:2182,localhost:2183 --topic test
> Group Topic Pid Offset logSize
> Lag Owner
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
> bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094
> --topic test
> THis ia s test message
> [2014-02-22 14:16:26,924] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,120] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,259] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,293] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,443] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,499] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,630] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,693] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,881] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
>
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
>
> kafka.common.FailedToSendMessageException: Failed to send messages after
> 3 tries.
> at kafka.producer.async.DefaultEventHandler.handle(
> DefaultEventHandler.scala:90)
> at kafka.producer.async.ProducerSendThread.tryToHandle(
> ProducerSendThread.scala:104)
> at kafka.producer.async.ProducerSendThread$$anonfun$
> processEvents$3.apply(ProducerSendThread.scala:87)
> at kafka.producer.async.ProducerSendThread$$anonfun$
> processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at kafka.producer.async.ProducerSendThread.processEvents(
> ProducerSendThread.scala:66)
> at kafka.producer.async.ProducerSendThread.run(
> ProducerSendThread.scala:44)
>
>
> arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-list-topic.sh
> --zookeeper localhost:2181
> topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
> topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
> topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
> topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
> topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
> topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
> topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
> topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
> topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
> topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
> topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
> topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
>
>
> What i did:
> started 3 kafka processes in my system.The config is default config with
> just the ports and log dir's changed.There are 3 zookeeper servers running.
> If i start the console producer next time it will work fine.
> To reproduce what did was, stopped all the kafka and zookeeper process,
> deleted the log dirs of the kafka and the zookeeper and restated the
> processes.
>
>
> Thanks
> Arjun Narasimha Kota
>
>
>
> On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
>
>> Could you do a list topic and show the output? Also, any error in the
>> controller and state-change log?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>>
>> hi,
>>>
>>> I am testing kafka 0.8 on my local machine. i have only one zookeeper and
>>> one kafka broker running.
>>>
>>> when i run the console producer i get this error:
>>>
>>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>>> partition due to: Failed to fetch topic metadata for topic: test
>>> (kafka.producer.async.DefaultEventHandler)
>>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>>> [{TopicMetadata for topic test ->
>>> No partition metadata for topic test due to kafka.common.
>>> LeaderNotAvailableException}]
>>> for topic [test]: class kafka.common.LeaderNotAvailableException
>>> (kafka.producer.BrokerPartitionInfo)
>>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>>> ProducerSendThread)
>>> kafka.common.FailedToSendMessageException: Failed to send messages after
>>> 3 tries.
>>> at kafka.producer.async.DefaultEventHandler.handle(
>>> DefaultEventHandler.scala:90)
>>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>>> ProducerSendThread.scala:104)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:87)
>>> at kafka.producer.async.ProducerSendThread$$anonfun$
>>> processEvents$3.apply(ProducerSendThread.scala:67)
>>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>> at kafka.producer.async.ProducerSendThread.processEvents(
>>> ProducerSendThread.scala:66)
>>> at kafka.producer.async.ProducerSendThread.run(
>>> ProducerSendThread.scala:44)
>>>
>>> Can some one please let me know why this is happening.
>>> I checked my vhost files and localhost, my hostname are correct.
>>>
>>>
>
Re: producer not pushing the data into broker
Posted by Arjun <ar...@socialtwist.com>.
Hi please find below the o/p of the list and the console producer. There
are no errors in the state change log or in the controller log.
One thing i found is this one happens only at starting time. I mean the
first time, when i start kafka and try to insert the messages i see this
error.
arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect
localhost:2181,localhost:2182,localhost:2183 --topic test
Group Topic Pid Offset
logSize Lag Owner
arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
bin/kafka-console-producer.sh --broker-list
localhost:9092,localhost:9093,localhost:9094 --topic test
THis ia s test message
[2014-02-22 14:16:26,924] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,120] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,121] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-02-22 14:16:27,259] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,293] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,295] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-02-22 14:16:27,443] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,499] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,499] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-02-22 14:16:27,630] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,693] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,694] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-02-22 14:16:27,881] WARN Error while fetching metadata
[{TopicMetadata for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-02-22 14:16:27,884] ERROR Failed to send requests for topics test
with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-02-22 14:16:27,886] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3 tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
arjunn@PRINHYLTPDL0061:~/Kafka/kafka_2.8.0-0.8.0$
bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: test partition: 0 leader: 2 replicas: 2,0 isr: 2,0
topic: test partition: 1 leader: 0 replicas: 0,1 isr: 0,1
topic: test partition: 2 leader: 1 replicas: 1,2 isr: 1,2
topic: test partition: 3 leader: 2 replicas: 2,1 isr: 2,1
topic: test partition: 4 leader: 0 replicas: 0,2 isr: 0,2
topic: test partition: 5 leader: 1 replicas: 1,0 isr: 1,0
topic: test partition: 6 leader: 2 replicas: 2,0 isr: 2,0
topic: test partition: 7 leader: 0 replicas: 0,1 isr: 0,1
topic: test partition: 8 leader: 1 replicas: 1,2 isr: 1,2
topic: test partition: 9 leader: 2 replicas: 2,1 isr: 2,1
topic: test partition: 10 leader: 0 replicas: 0,2 isr: 0,2
topic: test partition: 11 leader: 1 replicas: 1,0 isr: 1,0
What i did:
started 3 kafka processes in my system.The config is default config
with just the ports and log dir's changed.There are 3 zookeeper servers
running.
If i start the console producer next time it will work fine.
To reproduce what did was, stopped all the kafka and zookeeper process,
deleted the log dirs of the kafka and the zookeeper and restated the
processes.
Thanks
Arjun Narasimha Kota
On Friday 21 February 2014 09:41 PM, Jun Rao wrote:
> Could you do a list topic and show the output? Also, any error in the
> controller and state-change log?
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
>
>> hi,
>>
>> I am testing kafka 0.8 on my local machine. i have only one zookeeper and
>> one kafka broker running.
>>
>> when i run the console producer i get this error:
>>
>> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
>> partition due to: Failed to fetch topic metadata for topic: test
>> (kafka.producer.async.DefaultEventHandler)
>> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
>> [{TopicMetadata for topic test ->
>> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
>> for topic [test]: class kafka.common.LeaderNotAvailableException
>> (kafka.producer.BrokerPartitionInfo)
>> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
>> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
>> ERROR Error in handling batch of 1 events (kafka.producer.async.
>> ProducerSendThread)
>> kafka.common.FailedToSendMessageException: Failed to send messages after
>> 3 tries.
>> at kafka.producer.async.DefaultEventHandler.handle(
>> DefaultEventHandler.scala:90)
>> at kafka.producer.async.ProducerSendThread.tryToHandle(
>> ProducerSendThread.scala:104)
>> at kafka.producer.async.ProducerSendThread$$anonfun$
>> processEvents$3.apply(ProducerSendThread.scala:87)
>> at kafka.producer.async.ProducerSendThread$$anonfun$
>> processEvents$3.apply(ProducerSendThread.scala:67)
>> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>> at kafka.producer.async.ProducerSendThread.processEvents(
>> ProducerSendThread.scala:66)
>> at kafka.producer.async.ProducerSendThread.run(
>> ProducerSendThread.scala:44)
>>
>> Can some one please let me know why this is happening.
>> I checked my vhost files and localhost, my hostname are correct.
>>
Re: producer not pushing the data into broker
Posted by Jun Rao <ju...@gmail.com>.
Could you do a list topic and show the output? Also, any error in the
controller and state-change log?
Thanks,
Jun
On Fri, Feb 21, 2014 at 2:33 AM, Arjun <ar...@socialtwist.com> wrote:
> hi,
>
> I am testing kafka 0.8 on my local machine. i have only one zookeeper and
> one kafka broker running.
>
> when i run the console producer i get this error:
>
> [2014-02-21 16:01:20,512] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-21 16:01:20,513] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-02-21 16:01:20,632] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}]
> for topic [test]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-02-21 16:01:20,636] ERROR Failed to send requests for topics test
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> ERROR Error in handling batch of 1 events (kafka.producer.async.
> ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after
> 3 tries.
> at kafka.producer.async.DefaultEventHandler.handle(
> DefaultEventHandler.scala:90)
> at kafka.producer.async.ProducerSendThread.tryToHandle(
> ProducerSendThread.scala:104)
> at kafka.producer.async.ProducerSendThread$$anonfun$
> processEvents$3.apply(ProducerSendThread.scala:87)
> at kafka.producer.async.ProducerSendThread$$anonfun$
> processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at kafka.producer.async.ProducerSendThread.processEvents(
> ProducerSendThread.scala:66)
> at kafka.producer.async.ProducerSendThread.run(
> ProducerSendThread.scala:44)
>
> Can some one please let me know why this is happening.
> I checked my vhost files and localhost, my hostname are correct.
>