You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shafaq <s....@gmail.com> on 2013/10/10 01:30:07 UTC
Topic Fetch metadata in producer
I run into following error while doing some performance testing in Kafka
with producers running in multiple threads.
I can see the topic under /broker/topics in zooKeeper
I believe the producer tries to get the metadata info from Zookeeper.
I have tried to restart the 2-node Kafka broker cluster but of no use.
Any idea what can cause this to happen?
I'm using kafka 0.8 version
---------------------------------------------------------------------------------------
Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
to fetch topic metadata for topic: 37topic1381360774212
at
kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
--
Kind Regards,
Shafaq
Re: Topic Fetch metadata in producer
Posted by Neha Narkhede <ne...@gmail.com>.
Do you see the following warn messages on the producer ?
Fetching topic metadata with correlation id %d for topics [%s] from broker
[%s] failed
fetching topic metadata for topics [%s] from broker [%s] failed
On the server side -
1. could you enable DEBUG on kafka.server.KafkaApis and see if you get the
following message -
Error while fetching topic metadata for topic %s due to %s
2. Do you see the following ERROR message on the server -
"Error while fetching metadata for partition %s"
On Wed, Oct 9, 2013 at 10:57 PM, Shafaq <s....@gmail.com> wrote:
> I did some more debugging and found the corresponding debug message on the
> broker for topic (417topic1381383668416) in the producer:
>
> Seems like broker is removing fetcher for partition as producer throws no
> metadata for topic.
>
>
> [2013-10-09 22:41:09,529] INFO [Log Manager on Broker 2] Created log for
> partition [417topic1381383668416,3] in /tmp/kafka-logs-3.
> (kafka.log.LogManager)
> [2013-10-09 22:41:09,532] WARN No previously checkpointed highwatermark
> value found for topic 417topic1381383668416 partition 3. Returning 0 as the
> highwatermark (kafka.server.HighwaterMarkCheckpoint)
> [2013-10-09 22:41:09,533] INFO [ReplicaFetcherManager on broker 2] Removing
> fetcher for partition [417topic1381383668416,2]
> (kafka.server.ReplicaFetcherManager)
> [2013-10-09 22:41:09,534] INFO [Kafka Log on Broker 2], Completed load of
> log 417topic1381383668416-2 with log end offset 0 (kafka.log.Log)
> [2013-10-09 22:41:09,534] INFO [Log Manager on Broker 2] Created log for
> partition [417topic1381383668416,2] in /tmp/kafka-logs-3.
> (kafka.log.LogManager)
> [2013-10-09 22:41:09,537] WARN No previously checkpointed highwatermark
> value found for topic 417topic1381383668416 partition 2. Returning 0 as the
> highwatermark (kafka.server.HighwaterMarkCheckpoint)
> [2013-10-09 22:41:09,538] INFO [ReplicaFetcherManager on broker 2] Removing
> fetcher for partition [417topic1381383668416,4]
> (kafka.server.ReplicaFetcherManager)
> [2013-10-09 22:41:09,539] INFO [Kafka Log on Broker 2], Completed load of
> log 417topic1381383668416-4 with log end offset 0 (kafka.log.Log)
> [2013-10-09 22:41:09,539] INFO [Log Manager on Broker 2] Created log for
> partition [417topic1381383668416,4] in /tmp/kafka-logs-3.
> (kafka.log.LogManager)
> [2013-10-09 22:41:09,542] WARN No previously checkpointed highwatermark
> value found for topic 417topic1381383668416 partition 4. Returning 0 as the
> highwatermark (kafka.server.HighwaterMarkCheckpoint)
> [2013-10-09 22:41:09,543] INFO [ReplicaFetcherManager on broker 2] Removing
> fetcher for partition [417topic1381383668416,0]
> (kafka.server.ReplicaFetcherManager)
> [2013-10-09 22:41:09,544] INFO [Kafka Log on Broker 2], Completed load of
> log 417topic1381383668416-0 with log end offset 0 (kafka.log.Log)
> [2013-10-09 22:41:09,545] INFO [Log Manager on Broker 2] Created log for
> partition [417topic1381383668416,0] in /tmp/kafka-logs-3.
> (kafka.log.LogManager)
> [2013-10-09 22:41:09,547] WARN No previously checkpointed highwatermark
> value found for topic 417topic1381383668416 partition 0. Returning 0 as the
> highwatermark (kafka.server.HighwaterMarkCheckpoint)
> [2013-10-09 22:41:09,548] INFO [ReplicaFetcherManager on broker 2] Removing
> fetcher for partition [417topic1381383668416,1]
> (kafka.server.ReplicaFetcherManager)
> [2013-10-09 22:41:09,549] INFO [Kafka Log on Broker 2], Completed load of
> log 417topic1381383668416-1 with log end offset 0 (kafka.log.Log)
> [2013-10-09 22:41:09,549] INFO [Log Manager on Broker 2] Created log for
> partition [417topic1381383668416,1] in /tmp/kafka-logs-3.
> (kafka.log.LogManager)
> [2013-10-09 22:41:09,551] WARN No previously checkpointed highwatermark
> value found for topic 417topic1381383668416 partition 1. Returning 0 as the
> highwatermark (kafka.server.HighwaterMarkCheckpoint)
> [2013-10-09 22:41:09,552] INFO [Replica Manager on Broker 2]: Handled
> leader and isr request
>
> Name:LeaderAndIsrRequest;Version:0;Controller:2;ControllerEpoch:2;CorrelationId:18;ClientId:id_2-host_stg-analytics-002.int.funzio.com-port_9092;PartitionState:(417topic1381383668416,3)
> ->
>
> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,2)
> ->
>
> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,4)
> ->
>
> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,0)
> ->
>
> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,1)
> ->
>
> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2);Leaders:id:2,host:
> stg-analytics-002.int.funzio.com,port:9092 (kafka.server.ReplicaManager)
> java.io.IOException: No such file or directory
>
>
>
> On Wed, Oct 9, 2013 at 10:25 PM, Shafaq <s....@gmail.com> wrote:
>
> > I could not see such messages (Error while fetching metadata [%s] for
> > topic [%s]: %s ")
> >
> > In the perf test, I create new topic for sending the JSON blob in a new
> > producer thread.
> >
> > The error happened when I increased no. of files (mapping to producer
> > threads). There are already topics of previous performance test present
> in
> > the kafka cluster having 0.8 head version.
> >
> >
> > Regards,
> > S.Abdullah
> >
> >
> >
> > On Wed, Oct 9, 2013 at 6:08 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Do you see WARN messages like -
> >>
> >> "Error while fetching metadata [%s] for topic [%s]: %s "
> >> OR
> >> "Error while fetching metadata %s for topic partition [%s,%d]: [%s]"
> >>
> >>
> >> On Wed, Oct 9, 2013 at 5:49 PM, Shafaq <s....@gmail.com> wrote:
> >>
> >> > yes I'm using 0.8 head
> >> >
> >> >
> >> > roducer starting with option to multi-thread
> >> > ----compressed---true
> >> > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
> >> Failed
> >> > to fetch topic metadata for topic: 225topic1381362148396
> >> > at
> >> >
> >> >
> >>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> >> > at
> >> >
> >> >
> >>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> >> > at
> >> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> >> > at kafka.producer.Producer.send(Producer.scala:74)
> >> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> >> > at
> >> >
> >> >
> >>
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> >> > at
> >> >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> > at
> >> >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> > at java.lang.Thread.run(Thread.java:724)
> >> >
> >> >
> >> > ----Processing file multithreaded
> >> > mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz
> >> with
> >> > compression enabled = true
> >> > Exception in thread "pool-1-thread-2" kafka.common.KafkaException:
> >> Failed
> >> > to fetch topic metadata for topic: 166topic1381362148385
> >> > at
> >> >
> >> >
> >>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> >> > at
> >> >
> >> >
> >>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> >> > at
> >> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> >> > at
> >> >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> >> > at kafka.producer.Producer.send(Producer.scala:74)
> >> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> >> > at
> >> >
> >> >
> >>
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> >> > at
> >> >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> > at
> >> >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> > at java.lang.Thread.run(Thread.java:724)
> >> >
> >> >
> >> > On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> >> > >wrote:
> >> >
> >> > > Are you using 0.8 HEAD ?
> >> > >
> >> > > Can you send around the full stack trace since one of the common
> >> reasons
> >> > > for failed topic metadata requests are socket timeouts.
> >> > >
> >> > > Thanks,
> >> > > Neha
> >> > >
> >> > >
> >> > > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com>
> >> wrote:
> >> > >
> >> > > > I run into following error while doing some performance testing in
> >> > Kafka
> >> > > > with producers running in multiple threads.
> >> > > >
> >> > > > I can see the topic under /broker/topics in zooKeeper
> >> > > >
> >> > > > I believe the producer tries to get the metadata info from
> >> Zookeeper.
> >> > > >
> >> > > > I have tried to restart the 2-node Kafka broker cluster but of no
> >> use.
> >> > > >
> >> > > > Any idea what can cause this to happen?
> >> > > > I'm using kafka 0.8 version
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> ---------------------------------------------------------------------------------------
> >> > > >
> >> > > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
> >> > Failed
> >> > > > to fetch topic metadata for topic: 37topic1381360774212
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> >> > > > --
> >> > > > Kind Regards,
> >> > > > Shafaq
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Kind Regards,
> >> > Shafaq
> >> >
> >>
> >
> >
> >
> > --
> > Kind Regards,
> > Shafaq
> >
> >
>
>
> --
> Kind Regards,
> Shafaq
>
Re: Topic Fetch metadata in producer
Posted by Shafaq <s....@gmail.com>.
I did some more debugging and found the corresponding debug message on the
broker for topic (417topic1381383668416) in the producer:
Seems like broker is removing fetcher for partition as producer throws no
metadata for topic.
[2013-10-09 22:41:09,529] INFO [Log Manager on Broker 2] Created log for
partition [417topic1381383668416,3] in /tmp/kafka-logs-3.
(kafka.log.LogManager)
[2013-10-09 22:41:09,532] WARN No previously checkpointed highwatermark
value found for topic 417topic1381383668416 partition 3. Returning 0 as the
highwatermark (kafka.server.HighwaterMarkCheckpoint)
[2013-10-09 22:41:09,533] INFO [ReplicaFetcherManager on broker 2] Removing
fetcher for partition [417topic1381383668416,2]
(kafka.server.ReplicaFetcherManager)
[2013-10-09 22:41:09,534] INFO [Kafka Log on Broker 2], Completed load of
log 417topic1381383668416-2 with log end offset 0 (kafka.log.Log)
[2013-10-09 22:41:09,534] INFO [Log Manager on Broker 2] Created log for
partition [417topic1381383668416,2] in /tmp/kafka-logs-3.
(kafka.log.LogManager)
[2013-10-09 22:41:09,537] WARN No previously checkpointed highwatermark
value found for topic 417topic1381383668416 partition 2. Returning 0 as the
highwatermark (kafka.server.HighwaterMarkCheckpoint)
[2013-10-09 22:41:09,538] INFO [ReplicaFetcherManager on broker 2] Removing
fetcher for partition [417topic1381383668416,4]
(kafka.server.ReplicaFetcherManager)
[2013-10-09 22:41:09,539] INFO [Kafka Log on Broker 2], Completed load of
log 417topic1381383668416-4 with log end offset 0 (kafka.log.Log)
[2013-10-09 22:41:09,539] INFO [Log Manager on Broker 2] Created log for
partition [417topic1381383668416,4] in /tmp/kafka-logs-3.
(kafka.log.LogManager)
[2013-10-09 22:41:09,542] WARN No previously checkpointed highwatermark
value found for topic 417topic1381383668416 partition 4. Returning 0 as the
highwatermark (kafka.server.HighwaterMarkCheckpoint)
[2013-10-09 22:41:09,543] INFO [ReplicaFetcherManager on broker 2] Removing
fetcher for partition [417topic1381383668416,0]
(kafka.server.ReplicaFetcherManager)
[2013-10-09 22:41:09,544] INFO [Kafka Log on Broker 2], Completed load of
log 417topic1381383668416-0 with log end offset 0 (kafka.log.Log)
[2013-10-09 22:41:09,545] INFO [Log Manager on Broker 2] Created log for
partition [417topic1381383668416,0] in /tmp/kafka-logs-3.
(kafka.log.LogManager)
[2013-10-09 22:41:09,547] WARN No previously checkpointed highwatermark
value found for topic 417topic1381383668416 partition 0. Returning 0 as the
highwatermark (kafka.server.HighwaterMarkCheckpoint)
[2013-10-09 22:41:09,548] INFO [ReplicaFetcherManager on broker 2] Removing
fetcher for partition [417topic1381383668416,1]
(kafka.server.ReplicaFetcherManager)
[2013-10-09 22:41:09,549] INFO [Kafka Log on Broker 2], Completed load of
log 417topic1381383668416-1 with log end offset 0 (kafka.log.Log)
[2013-10-09 22:41:09,549] INFO [Log Manager on Broker 2] Created log for
partition [417topic1381383668416,1] in /tmp/kafka-logs-3.
(kafka.log.LogManager)
[2013-10-09 22:41:09,551] WARN No previously checkpointed highwatermark
value found for topic 417topic1381383668416 partition 1. Returning 0 as the
highwatermark (kafka.server.HighwaterMarkCheckpoint)
[2013-10-09 22:41:09,552] INFO [Replica Manager on Broker 2]: Handled
leader and isr request
Name:LeaderAndIsrRequest;Version:0;Controller:2;ControllerEpoch:2;CorrelationId:18;ClientId:id_2-host_stg-analytics-002.int.funzio.com-port_9092;PartitionState:(417topic1381383668416,3)
->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,2)
->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,4)
->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,0)
->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2),(417topic1381383668416,1)
->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:2);Leaders:id:2,host:
stg-analytics-002.int.funzio.com,port:9092 (kafka.server.ReplicaManager)
java.io.IOException: No such file or directory
On Wed, Oct 9, 2013 at 10:25 PM, Shafaq <s....@gmail.com> wrote:
> I could not see such messages (Error while fetching metadata [%s] for
> topic [%s]: %s ")
>
> In the perf test, I create new topic for sending the JSON blob in a new
> producer thread.
>
> The error happened when I increased no. of files (mapping to producer
> threads). There are already topics of previous performance test present in
> the kafka cluster having 0.8 head version.
>
>
> Regards,
> S.Abdullah
>
>
>
> On Wed, Oct 9, 2013 at 6:08 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Do you see WARN messages like -
>>
>> "Error while fetching metadata [%s] for topic [%s]: %s "
>> OR
>> "Error while fetching metadata %s for topic partition [%s,%d]: [%s]"
>>
>>
>> On Wed, Oct 9, 2013 at 5:49 PM, Shafaq <s....@gmail.com> wrote:
>>
>> > yes I'm using 0.8 head
>> >
>> >
>> > roducer starting with option to multi-thread
>> > ----compressed---true
>> > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
>> Failed
>> > to fetch topic metadata for topic: 225topic1381362148396
>> > at
>> >
>> >
>> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
>> > at
>> >
>> >
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
>> > at
>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
>> > at kafka.producer.Producer.send(Producer.scala:74)
>> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
>> > at
>> >
>> >
>> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:724)
>> >
>> >
>> > ----Processing file multithreaded
>> > mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz
>> with
>> > compression enabled = true
>> > Exception in thread "pool-1-thread-2" kafka.common.KafkaException:
>> Failed
>> > to fetch topic metadata for topic: 166topic1381362148385
>> > at
>> >
>> >
>> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
>> > at
>> >
>> >
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
>> > at
>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
>> > at
>> >
>> >
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
>> > at kafka.producer.Producer.send(Producer.scala:74)
>> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
>> > at
>> >
>> >
>> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:724)
>> >
>> >
>> > On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <neha.narkhede@gmail.com
>> > >wrote:
>> >
>> > > Are you using 0.8 HEAD ?
>> > >
>> > > Can you send around the full stack trace since one of the common
>> reasons
>> > > for failed topic metadata requests are socket timeouts.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com>
>> wrote:
>> > >
>> > > > I run into following error while doing some performance testing in
>> > Kafka
>> > > > with producers running in multiple threads.
>> > > >
>> > > > I can see the topic under /broker/topics in zooKeeper
>> > > >
>> > > > I believe the producer tries to get the metadata info from
>> Zookeeper.
>> > > >
>> > > > I have tried to restart the 2-node Kafka broker cluster but of no
>> use.
>> > > >
>> > > > Any idea what can cause this to happen?
>> > > > I'm using kafka 0.8 version
>> > > >
>> > > >
>> > >
>> >
>> ---------------------------------------------------------------------------------------
>> > > >
>> > > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
>> > Failed
>> > > > to fetch topic metadata for topic: 37topic1381360774212
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
>> > > > --
>> > > > Kind Regards,
>> > > > Shafaq
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Kind Regards,
>> > Shafaq
>> >
>>
>
>
>
> --
> Kind Regards,
> Shafaq
>
>
--
Kind Regards,
Shafaq
Re: Topic Fetch metadata in producer
Posted by Shafaq <s....@gmail.com>.
I could not see such messages (Error while fetching metadata [%s] for topic
[%s]: %s ")
In the perf test, I create new topic for sending the JSON blob in a new
producer thread.
The error happened when I increased no. of files (mapping to producer
threads). There are already topics of previous performance test present in
the kafka cluster having 0.8 head version.
Regards,
S.Abdullah
On Wed, Oct 9, 2013 at 6:08 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Do you see WARN messages like -
>
> "Error while fetching metadata [%s] for topic [%s]: %s "
> OR
> "Error while fetching metadata %s for topic partition [%s,%d]: [%s]"
>
>
> On Wed, Oct 9, 2013 at 5:49 PM, Shafaq <s....@gmail.com> wrote:
>
> > yes I'm using 0.8 head
> >
> >
> > roducer starting with option to multi-thread
> > ----compressed---true
> > Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
> > to fetch topic metadata for topic: 225topic1381362148396
> > at
> >
> >
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> > at
> >
> >
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> > at kafka.producer.Producer.send(Producer.scala:74)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> > at
> >
> >
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:724)
> >
> >
> > ----Processing file multithreaded
> > mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz
> with
> > compression enabled = true
> > Exception in thread "pool-1-thread-2" kafka.common.KafkaException: Failed
> > to fetch topic metadata for topic: 166topic1381362148385
> > at
> >
> >
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> > at
> >
> >
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> > at kafka.producer.Producer.send(Producer.scala:74)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> > at
> >
> >
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:724)
> >
> >
> > On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Are you using 0.8 HEAD ?
> > >
> > > Can you send around the full stack trace since one of the common
> reasons
> > > for failed topic metadata requests are socket timeouts.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com>
> wrote:
> > >
> > > > I run into following error while doing some performance testing in
> > Kafka
> > > > with producers running in multiple threads.
> > > >
> > > > I can see the topic under /broker/topics in zooKeeper
> > > >
> > > > I believe the producer tries to get the metadata info from Zookeeper.
> > > >
> > > > I have tried to restart the 2-node Kafka broker cluster but of no
> use.
> > > >
> > > > Any idea what can cause this to happen?
> > > > I'm using kafka 0.8 version
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------------------------
> > > >
> > > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
> > Failed
> > > > to fetch topic metadata for topic: 37topic1381360774212
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> > > > --
> > > > Kind Regards,
> > > > Shafaq
> > > >
> > >
> >
> >
> >
> > --
> > Kind Regards,
> > Shafaq
> >
>
--
Kind Regards,
Shafaq
Re: Topic Fetch metadata in producer
Posted by Neha Narkhede <ne...@gmail.com>.
Do you see WARN messages like -
"Error while fetching metadata [%s] for topic [%s]: %s "
OR
"Error while fetching metadata %s for topic partition [%s,%d]: [%s]"
On Wed, Oct 9, 2013 at 5:49 PM, Shafaq <s....@gmail.com> wrote:
> yes I'm using 0.8 head
>
>
> roducer starting with option to multi-thread
> ----compressed---true
> Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
> to fetch topic metadata for topic: 225topic1381362148396
> at
>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> at
>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> at
>
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> at
>
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:74)
> at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> at
>
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
>
> ----Processing file multithreaded
> mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz with
> compression enabled = true
> Exception in thread "pool-1-thread-2" kafka.common.KafkaException: Failed
> to fetch topic metadata for topic: 166topic1381362148385
> at
>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> at
>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
> at
>
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
> at
>
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
> at
>
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:74)
> at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> at
>
> gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
>
> On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Are you using 0.8 HEAD ?
> >
> > Can you send around the full stack trace since one of the common reasons
> > for failed topic metadata requests are socket timeouts.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com> wrote:
> >
> > > I run into following error while doing some performance testing in
> Kafka
> > > with producers running in multiple threads.
> > >
> > > I can see the topic under /broker/topics in zooKeeper
> > >
> > > I believe the producer tries to get the metadata info from Zookeeper.
> > >
> > > I have tried to restart the 2-node Kafka broker cluster but of no use.
> > >
> > > Any idea what can cause this to happen?
> > > I'm using kafka 0.8 version
> > >
> > >
> >
> ---------------------------------------------------------------------------------------
> > >
> > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException:
> Failed
> > > to fetch topic metadata for topic: 37topic1381360774212
> > > at
> > >
> > >
> >
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> > > at
> > >
> > >
> >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> > > --
> > > Kind Regards,
> > > Shafaq
> > >
> >
>
>
>
> --
> Kind Regards,
> Shafaq
>
Re: Topic Fetch metadata in producer
Posted by Shafaq <s....@gmail.com>.
yes I'm using 0.8 head
roducer starting with option to multi-thread
----compressed---true
Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
to fetch topic metadata for topic: 225topic1381362148396
at
kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
at
kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
at
kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:74)
at kafka.javaapi.producer.Producer.send(Producer.scala:32)
at
gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
----Processing file multithreaded
mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz with
compression enabled = true
Exception in thread "pool-1-thread-2" kafka.common.KafkaException: Failed
to fetch topic metadata for topic: 166topic1381362148385
at
kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
at
kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
at
kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
at
kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:74)
at kafka.javaapi.producer.Producer.send(Producer.scala:32)
at
gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Are you using 0.8 HEAD ?
>
> Can you send around the full stack trace since one of the common reasons
> for failed topic metadata requests are socket timeouts.
>
> Thanks,
> Neha
>
>
> On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com> wrote:
>
> > I run into following error while doing some performance testing in Kafka
> > with producers running in multiple threads.
> >
> > I can see the topic under /broker/topics in zooKeeper
> >
> > I believe the producer tries to get the metadata info from Zookeeper.
> >
> > I have tried to restart the 2-node Kafka broker cluster but of no use.
> >
> > Any idea what can cause this to happen?
> > I'm using kafka 0.8 version
> >
> >
> ---------------------------------------------------------------------------------------
> >
> > Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
> > to fetch topic metadata for topic: 37topic1381360774212
> > at
> >
> >
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> > --
> > Kind Regards,
> > Shafaq
> >
>
--
Kind Regards,
Shafaq
Re: Topic Fetch metadata in producer
Posted by Neha Narkhede <ne...@gmail.com>.
Are you using 0.8 HEAD ?
Can you send around the full stack trace since one of the common reasons
for failed topic metadata requests are socket timeouts.
Thanks,
Neha
On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s....@gmail.com> wrote:
> I run into following error while doing some performance testing in Kafka
> with producers running in multiple threads.
>
> I can see the topic under /broker/topics in zooKeeper
>
> I believe the producer tries to get the metadata info from Zookeeper.
>
> I have tried to restart the 2-node Kafka broker cluster but of no use.
>
> Any idea what can cause this to happen?
> I'm using kafka 0.8 version
>
> ---------------------------------------------------------------------------------------
>
> Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed
> to fetch topic metadata for topic: 37topic1381360774212
> at
>
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53)
> at
>
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185)
> --
> Kind Regards,
> Shafaq
>