You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Monika Garg <ga...@gmail.com> on 2013/10/17 10:51:14 UTC
getting error in fetching topic metadata,when giving message to
console producer in Kafka cluster
Hi,
I had 3 nodes Kafka cluster with default.replication.factor=3 in all the
three nodes.
I create a topic "topic1" with 7 partitions and with rep-factor = 3,my list
topic command gave below o/p:
topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr: 1,2,3
topic: topic1 partition: 1 leader: 2 replicas: 1,3,2 isr: 2,1,3
topic: topic1 partition: 2 leader: 3 replicas: 2,1,3 isr: 3,1,2
topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr: 1,2,3
topic: topic1 partition: 4 leader: 2 replicas: 1,2,3 isr: 2,1,3
topic: topic1 partition: 5 leader: 3 replicas: 2,3,1 isr: 3,1,2
topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr: 1,2,3
Then I removed 2 nodes.
Now I have single node cluster with the same property
default.replication.factor=3.So the list topic command is giving me the
below o/p:
topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr: 1
topic: topic1 partition: 1 leader: 1 replicas: 1,3,2 isr: 1
topic: topic1 partition: 2 leader: 1 replicas: 2,1,3 isr: 1
topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr: 1
topic: topic1 partition: 4 leader: 1 replicas: 1,2,3 isr: 1
topic: topic1 partition: 5 leader: 1 replicas: 2,3,1 isr: 1
topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr: 1
Now I started consoleProducer and give some message ,it is giving me
Warning
WARN Fetching topic metadata with correlation id 8 for topics [Set(topic1)]
from broker [id:0,host:192.168.145.107,port:2181] failed
(kafka.client.ClientUtils$)
pasting here the full stack trace:
[2013-10-17 13:01:25,776] WARN Fetching topic metadata with correlation id
8 for topics [Set(topic2)] from broker
[id:0,host:192.168.145.107,port:2181] failed (kafka.client.ClientUtils$)
java.io.EOFException: Received -1 when reading from channel, socket has
likely been closed.
at kafka.utils.Utils$.read(Utils.scala:395)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at
kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at
kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
at kafka.utils.Utils$.swallow(Utils.scala:186)
at kafka.utils.Logging$class.swallowError(Logging.scala:105)
at kafka.utils.Utils$.swallowError(Utils.scala:45)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2013-10-17 13:01:25,778] ERROR fetching topic metadata for topics
[Set(topic2)] from broker
[ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
(kafka.utils.Utils$)
kafka.common.KafkaException: fetching topic metadata for topics
[Set(topic2)] from broker
[ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:67)
at
kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at
kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
at kafka.utils.Utils$.swallow(Utils.scala:186)
at kafka.utils.Logging$class.swallowError(Logging.scala:105)
at kafka.utils.Utils$.swallowError(Utils.scala:45)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
Caused by: java.io.EOFException: Received -1 when reading from channel,
socket has likely been closed.
at kafka.utils.Utils$.read(Utils.scala:395)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
... 12 more
[2013-10-17 13:01:25,787] ERROR Failed to send requests for topics topic2
with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2013-10-17 13:01:25,794] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
I am not getting the reason for this behaviour of consoleProducer.
--
*Moniii*
Re: getting error in fetching topic metadata,when giving message to
console producer in Kafka cluster
Posted by Monika Garg <ga...@gmail.com>.
Thanks Jun...You are right,bymistakenly I have given zookeeper port.
Now its working fine.
On Thu, Oct 17, 2013 at 8:50 PM, Jun Rao <ju...@gmail.com> wrote:
> I think you provided the wrong port in broker-list. 2181 is the ZK port.
> Kafka port defaults to 9092.
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 17, 2013 at 1:51 AM, Monika Garg <ga...@gmail.com> wrote:
>
> > Hi,
> >
> > I had 3 nodes Kafka cluster with default.replication.factor=3 in all the
> > three nodes.
> > I create a topic "topic1" with 7 partitions and with rep-factor = 3,my
> list
> > topic command gave below o/p:
> >
> > topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr:
> 1,2,3
> > topic: topic1 partition: 1 leader: 2 replicas: 1,3,2 isr:
> 2,1,3
> > topic: topic1 partition: 2 leader: 3 replicas: 2,1,3 isr:
> 3,1,2
> > topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr:
> 1,2,3
> > topic: topic1 partition: 4 leader: 2 replicas: 1,2,3 isr:
> 2,1,3
> > topic: topic1 partition: 5 leader: 3 replicas: 2,3,1 isr:
> 3,1,2
> > topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr:
> 1,2,3
> >
> >
> > Then I removed 2 nodes.
> >
> > Now I have single node cluster with the same property
> > default.replication.factor=3.So the list topic command is giving me the
> > below o/p:
> >
> > topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr: 1
> > topic: topic1 partition: 1 leader: 1 replicas: 1,3,2 isr: 1
> > topic: topic1 partition: 2 leader: 1 replicas: 2,1,3 isr: 1
> > topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr: 1
> > topic: topic1 partition: 4 leader: 1 replicas: 1,2,3 isr: 1
> > topic: topic1 partition: 5 leader: 1 replicas: 2,3,1 isr: 1
> > topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr: 1
> >
> > Now I started consoleProducer and give some message ,it is giving me
> > Warning
> > WARN Fetching topic metadata with correlation id 8 for topics
> [Set(topic1)]
> > from broker [id:0,host:192.168.145.107,port:2181] failed
> > (kafka.client.ClientUtils$)
> >
> > pasting here the full stack trace:
> >
> > [2013-10-17 13:01:25,776] WARN Fetching topic metadata with correlation
> id
> > 8 for topics [Set(topic2)] from broker
> > [id:0,host:192.168.145.107,port:2181] failed (kafka.client.ClientUtils$)
> > java.io.EOFException: Received -1 when reading from channel, socket has
> > likely been closed.
> > at kafka.utils.Utils$.read(Utils.scala:395)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
> > at
> >
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > at
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
> > at kafka.utils.Utils$.swallow(Utils.scala:186)
> > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > at kafka.utils.Utils$.swallowError(Utils.scala:45)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > [2013-10-17 13:01:25,778] ERROR fetching topic metadata for topics
> > [Set(topic2)] from broker
> > [ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
> > (kafka.utils.Utils$)
> > kafka.common.KafkaException: fetching topic metadata for topics
> > [Set(topic2)] from broker
> > [ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:67)
> > at
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
> > at kafka.utils.Utils$.swallow(Utils.scala:186)
> > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > at kafka.utils.Utils$.swallowError(Utils.scala:45)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > Caused by: java.io.EOFException: Received -1 when reading from channel,
> > socket has likely been closed.
> > at kafka.utils.Utils$.read(Utils.scala:395)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
> > at
> >
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > ... 12 more
> > [2013-10-17 13:01:25,787] ERROR Failed to send requests for topics topic2
> > with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> > [2013-10-17 13:01:25,794] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> >
> >
> >
> > I am not getting the reason for this behaviour of consoleProducer.
> >
> >
> >
> >
> > --
> > *Moniii*
> >
>
--
*Moniii*
Re: getting error in fetching topic metadata,when giving message to
console producer in Kafka cluster
Posted by Jun Rao <ju...@gmail.com>.
I think you provided the wrong port in broker-list. 2181 is the ZK port.
Kafka port defaults to 9092.
Thanks,
Jun
On Thu, Oct 17, 2013 at 1:51 AM, Monika Garg <ga...@gmail.com> wrote:
> Hi,
>
> I had 3 nodes Kafka cluster with default.replication.factor=3 in all the
> three nodes.
> I create a topic "topic1" with 7 partitions and with rep-factor = 3,my list
> topic command gave below o/p:
>
> topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr: 1,2,3
> topic: topic1 partition: 1 leader: 2 replicas: 1,3,2 isr: 2,1,3
> topic: topic1 partition: 2 leader: 3 replicas: 2,1,3 isr: 3,1,2
> topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr: 1,2,3
> topic: topic1 partition: 4 leader: 2 replicas: 1,2,3 isr: 2,1,3
> topic: topic1 partition: 5 leader: 3 replicas: 2,3,1 isr: 3,1,2
> topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr: 1,2,3
>
>
> Then I removed 2 nodes.
>
> Now I have single node cluster with the same property
> default.replication.factor=3.So the list topic command is giving me the
> below o/p:
>
> topic: topic1 partition: 0 leader: 1 replicas: 3,2,1 isr: 1
> topic: topic1 partition: 1 leader: 1 replicas: 1,3,2 isr: 1
> topic: topic1 partition: 2 leader: 1 replicas: 2,1,3 isr: 1
> topic: topic1 partition: 3 leader: 1 replicas: 3,1,2 isr: 1
> topic: topic1 partition: 4 leader: 1 replicas: 1,2,3 isr: 1
> topic: topic1 partition: 5 leader: 1 replicas: 2,3,1 isr: 1
> topic: topic1 partition: 6 leader: 1 replicas: 3,2,1 isr: 1
>
> Now I started consoleProducer and give some message ,it is giving me
> Warning
> WARN Fetching topic metadata with correlation id 8 for topics [Set(topic1)]
> from broker [id:0,host:192.168.145.107,port:2181] failed
> (kafka.client.ClientUtils$)
>
> pasting here the full stack trace:
>
> [2013-10-17 13:01:25,776] WARN Fetching topic metadata with correlation id
> 8 for topics [Set(topic2)] from broker
> [id:0,host:192.168.145.107,port:2181] failed (kafka.client.ClientUtils$)
> java.io.EOFException: Received -1 when reading from channel, socket has
> likely been closed.
> at kafka.utils.Utils$.read(Utils.scala:395)
> at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
> at
>
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
> at kafka.utils.Utils$.swallow(Utils.scala:186)
> at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> at kafka.utils.Utils$.swallowError(Utils.scala:45)
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> [2013-10-17 13:01:25,778] ERROR fetching topic metadata for topics
> [Set(topic2)] from broker
> [ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
> (kafka.utils.Utils$)
> kafka.common.KafkaException: fetching topic metadata for topics
> [Set(topic2)] from broker
> [ArrayBuffer(id:0,host:192.168.145.107,port:2181)] failed
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:67)
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
> at kafka.utils.Utils$.swallow(Utils.scala:186)
> at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> at kafka.utils.Utils$.swallowError(Utils.scala:45)
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> Caused by: java.io.EOFException: Received -1 when reading from channel,
> socket has likely been closed.
> at kafka.utils.Utils$.read(Utils.scala:395)
> at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
> at
>
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> ... 12 more
> [2013-10-17 13:01:25,787] ERROR Failed to send requests for topics topic2
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2013-10-17 13:01:25,794] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
>
>
>
> I am not getting the reason for this behaviour of consoleProducer.
>
>
>
>
> --
> *Moniii*
>