You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sumit Rangwala <su...@gmail.com> on 2015/02/04 09:58:16 UTC
Issue with auto topic creation as well
I am observing the following exception with kafka client:
2015-02-04 00:17:27,345
(LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread)
ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for
topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker
[id:49649,host:172.16.204.44,port:49649] failed
java.lang.ArrayIndexOutOfBoundsException: 7
at
kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.immutable.Range.foreach(Range.scala:81)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.Range.map(Range.scala:46)
at kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
# Background on what my code is doing:
In my setup kafka brokers are set for auto topic creation. In the scenario
above a node informs other nodes (currently 5 in total) about ~50 new
(non-existent) topics, and all the nodes almost simultaneously open a
consumer for each of these topics. This triggers topic creation for all the
topics on kafka brokers. Most of the topics are created fine but there is
almost always few topics that throw the above exception and a kafka
producer is unable to send any data to such a topic
(LAX1-GRIFFIN-r8-1423037468055
in the above case)
# Logs
All kafka broker logs (3 brokers) available at http://d.pr/f/1eOGM/5UPMPfg5
For these logs only LAX1-GRIFFIN-r8-1423037468055 had an issue. All other
topics were fine.
Setup
--------
Zookeeper: 3.4.6
Kafka broker: 0.8.2-beta
Kafka clients: 0.8.2-beta
# Kafka boker settings (all other settings are default 0.8.2-beta settings)
kafka.controlled.shutdown.enable: 'FALSE'
kafka.auto.create.topics.enable: 'TRUE'
kafka.num.partitions: 8
kafka.default.replication.factor: 1
kafka.rebalance.backoff.ms: 3000
kafka.rebalance.max.retries: 10
kafka.log.retention.minutes: 1200
kafka.delete.topic.enable: 'TRUE'
Sumit
Re: Issue with auto topic creation as well
Posted by Jun Rao <ju...@confluent.io>.
Hmm, anything special with that topic (e.g., did you run any admin
operation on it)? Also, does bouncing the brokers fix the issue?
Thanks,
Jun
On Tue, Feb 10, 2015 at 1:31 PM, Sumit Rangwala <su...@gmail.com>
wrote:
> On Thu, Feb 5, 2015 at 12:37 PM, Sumit Rangwala <su...@gmail.com>
> wrote:
>
> >
> >
> > On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> >> Could you try the 0.8.2.0 release? It fixed one issue related to topic
> >> creation.
> >>
> >
> Jun,
>
> If you need more info let me know. Seems like TopicMetadataResponse is
> expecting more fields then what is present in the response.
>
>
> Sumit
>
>
>
> >
> > Jun,
> >
> > Tried with 0.8.2.0 and I still see the same error.
> >
> > I see the error given below almost incessantly on the client side for
> > topic LAX1-GRIFFIN-r47-1423165897627. Things look fine one the broker
> side:
> > Topic:LAX1-GRIFFIN-r47-1423165897627 PartitionCount:8 ReplicationFactor:1
> > Configs:
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 0 Leader: 49817
> Replicas:
> > 49817 Isr: 49817
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 1 Leader: 49818
> Replicas:
> > 49818 Isr: 49818
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 2 Leader: 49814
> Replicas:
> > 49814 Isr: 49814
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 3 Leader: 49817
> Replicas:
> > 49817 Isr: 49817
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 4 Leader: 49818
> Replicas:
> > 49818 Isr: 49818
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 5 Leader: 49814
> Replicas:
> > 49814 Isr: 49814
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 6 Leader: 49817
> Replicas:
> > 49817 Isr: 49817
> > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 7 Leader: 49818
> Replicas:
> > 49818 Isr: 49818
> >
> >
> > Broker logs at http://d.pr/f/1jLOf/5H8uWPqu in case you need them
> >
> > ERROR:
> > 2015-02-05 12:02:03,764
> >
> (LAX1-GRIFFIN-r47-1423165897627-pf6898-lax1-GriffinDownloader-1423166442395_f27c6501f3d5-1423166444213-d572da8f-leader-finder-thread)
> > ClientUtils$ WARN: Fetching topic metadata with correlation id 152 for
> > topics [Set(LAX1-GRIFFIN-r47-1423165897627)] from broker
> > [id:49814,host:172.16.204.44,port:49814] failed
> > java.lang.ArrayIndexOutOfBoundsException: 7
> > at
> >
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> > at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> > at
> >
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> > at
> >
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> > at
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> > at
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> > at scala.collection.immutable.Range.foreach(Range.scala:81)
> > at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
> > at scala.collection.immutable.Range.map(Range.scala:46)
> > at
> > kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> > at
> >
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> >
> >
> >
> >
> >
> >
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala <
> sumitrangwala@gmail.com>
> >> wrote:
> >>
> >> > I am observing the following exception with kafka client:
> >> >
> >> > 2015-02-04 00:17:27,345
> >> >
> >> >
> >>
> (LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread)
> >> > ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for
> >> > topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker
> >> > [id:49649,host:172.16.204.44,port:49649] failed
> >> > java.lang.ArrayIndexOutOfBoundsException: 7
> >> > at
> >> >
> >> >
> >>
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> >> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> >> > at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> >> > at
> >> >
> >> >
> >>
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> >> > at
> >> >
> >> >
> >>
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> >> > at
> >> >
> >> >
> >>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> >> > at
> >> >
> >> >
> >>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> >> > at scala.collection.immutable.Range.foreach(Range.scala:81)
> >> > at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
> >> > at scala.collection.immutable.Range.map(Range.scala:46)
> >> > at
> >> >
> >>
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> >> > at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
> >> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> >> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> >> > at
> >> >
> >> >
> >>
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> >> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> >> >
> >> >
> >> >
> >> > # Background on what my code is doing:
> >> >
> >> > In my setup kafka brokers are set for auto topic creation. In the
> >> scenario
> >> > above a node informs other nodes (currently 5 in total) about ~50 new
> >> > (non-existent) topics, and all the nodes almost simultaneously open a
> >> > consumer for each of these topics. This triggers topic creation for
> all
> >> the
> >> > topics on kafka brokers. Most of the topics are created fine but there
> >> is
> >> > almost always few topics that throw the above exception and a kafka
> >> > producer is unable to send any data to such a topic
> >> > (LAX1-GRIFFIN-r8-1423037468055
> >> > in the above case)
> >> >
> >> >
> >> > # Logs
> >> > All kafka broker logs (3 brokers) available at
> >> > http://d.pr/f/1eOGM/5UPMPfg5
> >> > For these logs only LAX1-GRIFFIN-r8-1423037468055 had an issue. All
> >> other
> >> > topics were fine.
> >> >
> >> >
> >> > Setup
> >> > --------
> >> > Zookeeper: 3.4.6
> >> > Kafka broker: 0.8.2-beta
> >> > Kafka clients: 0.8.2-beta
> >> >
> >> > # Kafka boker settings (all other settings are default 0.8.2-beta
> >> settings)
> >> > kafka.controlled.shutdown.enable: 'FALSE'
> >> > kafka.auto.create.topics.enable: 'TRUE'
> >> > kafka.num.partitions: 8
> >> > kafka.default.replication.factor: 1
> >> > kafka.rebalance.backoff.ms: 3000
> >> > kafka.rebalance.max.retries: 10
> >> > kafka.log.retention.minutes: 1200
> >> > kafka.delete.topic.enable: 'TRUE'
> >> >
> >> >
> >> >
> >> > Sumit
> >> >
> >>
> >
> >
>
Re: Issue with auto topic creation as well
Posted by Sumit Rangwala <su...@gmail.com>.
On Thu, Feb 5, 2015 at 12:37 PM, Sumit Rangwala <su...@gmail.com>
wrote:
>
>
> On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao <ju...@confluent.io> wrote:
>
>> Could you try the 0.8.2.0 release? It fixed one issue related to topic
>> creation.
>>
>
Jun,
If you need more info let me know. Seems like TopicMetadataResponse is
expecting more fields then what is present in the response.
Sumit
>
> Jun,
>
> Tried with 0.8.2.0 and I still see the same error.
>
> I see the error given below almost incessantly on the client side for
> topic LAX1-GRIFFIN-r47-1423165897627. Things look fine one the broker side:
> Topic:LAX1-GRIFFIN-r47-1423165897627 PartitionCount:8 ReplicationFactor:1
> Configs:
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 0 Leader: 49817 Replicas:
> 49817 Isr: 49817
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 1 Leader: 49818 Replicas:
> 49818 Isr: 49818
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 2 Leader: 49814 Replicas:
> 49814 Isr: 49814
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 3 Leader: 49817 Replicas:
> 49817 Isr: 49817
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 4 Leader: 49818 Replicas:
> 49818 Isr: 49818
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 5 Leader: 49814 Replicas:
> 49814 Isr: 49814
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 6 Leader: 49817 Replicas:
> 49817 Isr: 49817
> Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 7 Leader: 49818 Replicas:
> 49818 Isr: 49818
>
>
> Broker logs at http://d.pr/f/1jLOf/5H8uWPqu in case you need them
>
> ERROR:
> 2015-02-05 12:02:03,764
> (LAX1-GRIFFIN-r47-1423165897627-pf6898-lax1-GriffinDownloader-1423166442395_f27c6501f3d5-1423166444213-d572da8f-leader-finder-thread)
> ClientUtils$ WARN: Fetching topic metadata with correlation id 152 for
> topics [Set(LAX1-GRIFFIN-r47-1423165897627)] from broker
> [id:49814,host:172.16.204.44,port:49814] failed
> java.lang.ArrayIndexOutOfBoundsException: 7
> at
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> at
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> at scala.collection.immutable.Range.foreach(Range.scala:81)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
> at scala.collection.immutable.Range.map(Range.scala:46)
> at
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
>
>
>
>
>
>
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala <su...@gmail.com>
>> wrote:
>>
>> > I am observing the following exception with kafka client:
>> >
>> > 2015-02-04 00:17:27,345
>> >
>> >
>> (LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread)
>> > ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for
>> > topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker
>> > [id:49649,host:172.16.204.44,port:49649] failed
>> > java.lang.ArrayIndexOutOfBoundsException: 7
>> > at
>> >
>> >
>> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
>> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
>> > at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
>> > at
>> >
>> >
>> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
>> > at
>> >
>> >
>> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
>> > at
>> >
>> >
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>> > at
>> >
>> >
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
>> > at scala.collection.immutable.Range.foreach(Range.scala:81)
>> > at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
>> > at scala.collection.immutable.Range.map(Range.scala:46)
>> > at
>> >
>> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
>> > at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
>> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
>> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
>> > at
>> >
>> >
>> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
>> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
>> >
>> >
>> >
>> > # Background on what my code is doing:
>> >
>> > In my setup kafka brokers are set for auto topic creation. In the
>> scenario
>> > above a node informs other nodes (currently 5 in total) about ~50 new
>> > (non-existent) topics, and all the nodes almost simultaneously open a
>> > consumer for each of these topics. This triggers topic creation for all
>> the
>> > topics on kafka brokers. Most of the topics are created fine but there
>> is
>> > almost always few topics that throw the above exception and a kafka
>> > producer is unable to send any data to such a topic
>> > (LAX1-GRIFFIN-r8-1423037468055
>> > in the above case)
>> >
>> >
>> > # Logs
>> > All kafka broker logs (3 brokers) available at
>> > http://d.pr/f/1eOGM/5UPMPfg5
>> > For these logs only LAX1-GRIFFIN-r8-1423037468055 had an issue. All
>> other
>> > topics were fine.
>> >
>> >
>> > Setup
>> > --------
>> > Zookeeper: 3.4.6
>> > Kafka broker: 0.8.2-beta
>> > Kafka clients: 0.8.2-beta
>> >
>> > # Kafka boker settings (all other settings are default 0.8.2-beta
>> settings)
>> > kafka.controlled.shutdown.enable: 'FALSE'
>> > kafka.auto.create.topics.enable: 'TRUE'
>> > kafka.num.partitions: 8
>> > kafka.default.replication.factor: 1
>> > kafka.rebalance.backoff.ms: 3000
>> > kafka.rebalance.max.retries: 10
>> > kafka.log.retention.minutes: 1200
>> > kafka.delete.topic.enable: 'TRUE'
>> >
>> >
>> >
>> > Sumit
>> >
>>
>
>
Re: Issue with auto topic creation as well
Posted by Sumit Rangwala <su...@gmail.com>.
On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao <ju...@confluent.io> wrote:
> Could you try the 0.8.2.0 release? It fixed one issue related to topic
> creation.
>
Jun,
Tried with 0.8.2.0 and I still see the same error.
I see the error given below almost incessantly on the client side for topic
LAX1-GRIFFIN-r47-1423165897627. Things look fine one the broker side:
Topic:LAX1-GRIFFIN-r47-1423165897627 PartitionCount:8 ReplicationFactor:1
Configs:
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 0 Leader: 49817 Replicas:
49817 Isr: 49817
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 1 Leader: 49818 Replicas:
49818 Isr: 49818
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 2 Leader: 49814 Replicas:
49814 Isr: 49814
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 3 Leader: 49817 Replicas:
49817 Isr: 49817
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 4 Leader: 49818 Replicas:
49818 Isr: 49818
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 5 Leader: 49814 Replicas:
49814 Isr: 49814
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 6 Leader: 49817 Replicas:
49817 Isr: 49817
Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 7 Leader: 49818 Replicas:
49818 Isr: 49818
Broker logs at http://d.pr/f/1jLOf/5H8uWPqu in case you need them
ERROR:
2015-02-05 12:02:03,764
(LAX1-GRIFFIN-r47-1423165897627-pf6898-lax1-GriffinDownloader-1423166442395_f27c6501f3d5-1423166444213-d572da8f-leader-finder-thread)
ClientUtils$ WARN: Fetching topic metadata with correlation id 152 for
topics [Set(LAX1-GRIFFIN-r47-1423165897627)] from broker
[id:49814,host:172.16.204.44,port:49814] failed
java.lang.ArrayIndexOutOfBoundsException: 7
at
kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.immutable.Range.foreach(Range.scala:81)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.Range.map(Range.scala:46)
at
kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
>
> Thanks,
>
> Jun
>
> On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala <su...@gmail.com>
> wrote:
>
> > I am observing the following exception with kafka client:
> >
> > 2015-02-04 00:17:27,345
> >
> >
> (LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread)
> > ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for
> > topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker
> > [id:49649,host:172.16.204.44,port:49649] failed
> > java.lang.ArrayIndexOutOfBoundsException: 7
> > at
> >
> >
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> > at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> > at
> >
> >
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> > at
> >
> >
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> > at
> >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> > at
> >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> > at scala.collection.immutable.Range.foreach(Range.scala:81)
> > at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
> > at scala.collection.immutable.Range.map(Range.scala:46)
> > at
> > kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> > at
> >
> >
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> >
> >
> >
> > # Background on what my code is doing:
> >
> > In my setup kafka brokers are set for auto topic creation. In the
> scenario
> > above a node informs other nodes (currently 5 in total) about ~50 new
> > (non-existent) topics, and all the nodes almost simultaneously open a
> > consumer for each of these topics. This triggers topic creation for all
> the
> > topics on kafka brokers. Most of the topics are created fine but there is
> > almost always few topics that throw the above exception and a kafka
> > producer is unable to send any data to such a topic
> > (LAX1-GRIFFIN-r8-1423037468055
> > in the above case)
> >
> >
> > # Logs
> > All kafka broker logs (3 brokers) available at
> > http://d.pr/f/1eOGM/5UPMPfg5
> > For these logs only LAX1-GRIFFIN-r8-1423037468055 had an issue. All other
> > topics were fine.
> >
> >
> > Setup
> > --------
> > Zookeeper: 3.4.6
> > Kafka broker: 0.8.2-beta
> > Kafka clients: 0.8.2-beta
> >
> > # Kafka boker settings (all other settings are default 0.8.2-beta
> settings)
> > kafka.controlled.shutdown.enable: 'FALSE'
> > kafka.auto.create.topics.enable: 'TRUE'
> > kafka.num.partitions: 8
> > kafka.default.replication.factor: 1
> > kafka.rebalance.backoff.ms: 3000
> > kafka.rebalance.max.retries: 10
> > kafka.log.retention.minutes: 1200
> > kafka.delete.topic.enable: 'TRUE'
> >
> >
> >
> > Sumit
> >
>
Re: Issue with auto topic creation as well
Posted by Jun Rao <ju...@confluent.io>.
Could you try the 0.8.2.0 release? It fixed one issue related to topic
creation.
Thanks,
Jun
On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala <su...@gmail.com>
wrote:
> I am observing the following exception with kafka client:
>
> 2015-02-04 00:17:27,345
>
> (LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread)
> ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for
> topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker
> [id:49649,host:172.16.204.44,port:49649] failed
> java.lang.ArrayIndexOutOfBoundsException: 7
> at
>
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> at
>
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at
>
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
> at scala.collection.immutable.Range.foreach(Range.scala:81)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
> at scala.collection.immutable.Range.map(Range.scala:46)
> at
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:115)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
> at
>
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
>
>
>
> # Background on what my code is doing:
>
> In my setup kafka brokers are set for auto topic creation. In the scenario
> above a node informs other nodes (currently 5 in total) about ~50 new
> (non-existent) topics, and all the nodes almost simultaneously open a
> consumer for each of these topics. This triggers topic creation for all the
> topics on kafka brokers. Most of the topics are created fine but there is
> almost always few topics that throw the above exception and a kafka
> producer is unable to send any data to such a topic
> (LAX1-GRIFFIN-r8-1423037468055
> in the above case)
>
>
> # Logs
> All kafka broker logs (3 brokers) available at
> http://d.pr/f/1eOGM/5UPMPfg5
> For these logs only LAX1-GRIFFIN-r8-1423037468055 had an issue. All other
> topics were fine.
>
>
> Setup
> --------
> Zookeeper: 3.4.6
> Kafka broker: 0.8.2-beta
> Kafka clients: 0.8.2-beta
>
> # Kafka boker settings (all other settings are default 0.8.2-beta settings)
> kafka.controlled.shutdown.enable: 'FALSE'
> kafka.auto.create.topics.enable: 'TRUE'
> kafka.num.partitions: 8
> kafka.default.replication.factor: 1
> kafka.rebalance.backoff.ms: 3000
> kafka.rebalance.max.retries: 10
> kafka.log.retention.minutes: 1200
> kafka.delete.topic.enable: 'TRUE'
>
>
>
> Sumit
>