You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "DashengJu (JIRA)" <ji...@apache.org> on 2014/07/02 11:12:09 UTC
[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException
Buffer.limit on FetchResponse.scala + 33
[ https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049766#comment-14049766 ]
DashengJu commented on KAFKA-1196:
----------------------------------
I have encounter this problem.
I have many topics and partitions, every topic have 2 replicas. Then one broker needs to consumer data from another broker. It seems the broker to fetch many replication, and the sync was failed, because of the error above. So the ISR list is always just the "preferred replica".
I think this is a big problem.
============= below is the error log ====================================
[2014-06-28 10:00:03,434] ERROR [ReplicaFetcherThread-0-1], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 710; ClientId: ReplicaFetcherThread-0-1; ReplicaId: 5; MaxWait: 3000 ms; MinBytes: 1 bytes; RequestInfo: [org.wasp_card_log,12] -> PartitionFetchInfo(0,67108864),[org.mobile_push_netevent,1] -> PartitionFetchInfo(492788,67108864),[org.mobile,8] -> PartitionFetchInfo(59767,67108864),[app.newbuyer,7] -> PartitionFetchInfo(42,67108864),[binlog.wwwdeal,6] -> PartitionFetchInfo(1718,67108864),[org.b,6] -> PartitionFetchInfo(150548,67108864),[org.filestorage,14] -> PartitionFetchInfo(11,67108864),[org.feedbackchange,6] -> PartitionFetchInfo(277,67108864),[org.eventlog,6] -> PartitionFetchInfo(101049,67108864),[org.bizappapi,10] -> PartitionFetchInfo(0,67108864),[org.data_resys_routerres,5] -> PartitionFetchInfo(606925,67108864),[log.mobile,5] -> PartitionFetchInfo(0,67108864),[org.mtrace,0] -> PartitionFetchInfo(75581,67108864),[org.mobile,3] -> PartitionFetchInfo(2062782,67108864),[org.data_resys_geo,10] -> PartitionFetchInfo(678,67108864),[org.data_asyncquerier_tablecolumn,12] -> PartitionFetchInfo(1,67108864),[org.mobile_push_applist,10] -> PartitionFetchInfo(2438334,67108864),[org.mobile_hotellog,12] -> PartitionFetchInfo(623,67108864),[org.mobile,15] -> PartitionFetchInfo(2376788,67108864),[org.stormlog,0] -> PartitionFetchInfo(0,67108864),[org.mtcrm,14] -> PartitionFetchInfo(250,67108864),[binlog.coupon,4] -> PartitionFetchInfo(149681,67108864),[org.mtrace,1] -> PartitionFetchInfo(1025277,67108864),[org.cos_sso,15] -> PartitionFetchInfo(3301,67108864),[org.data_resys_router,6] -> PartitionFetchInfo(1031,67108864),[log.mobile,3] -> PartitionFetchInfo(17453,67108864),[log.loginlog,7] -> PartitionFetchInfo(5932,67108864),[org.data_search_queryserver,13] -> PartitionFetchInfo(891,67108864),[org.mobile_push_applist,5] -> PartitionFetchInfo(180204,67108864),[org.data_resys_hotquery,4] -> PartitionFetchInfo(1153,67108864),[org.wasp_device,0] -> PartitionFetchInfo(10,67108864),[org.mobilerpc,9] -> PartitionFetchInfo(42137,67108864),[org.mtrace,9] -> PartitionFetchInfo(1783580,67108864),[org.mobile_push_applist,8] -> PartitionFetchInfo(2471375,67108864),[org.mobile_eventlog,1] -> PartitionFetchInfo(311414,67108864),[org.extendapply,15] -> PartitionFetchInfo(0,67108864),[app.newbuyer,14] -> PartitionFetchInfo(0,67108864),[org.mobile_push_record,13] -> PartitionFetchInfo(11,67108864),[org.mtrace,3] -> PartitionFetchInfo(1578588,67108864),[log.orderlog,8] -> PartitionFetchInfo(0,67108864),[org.mtrace,4] -> PartitionFetchInfo(122482,67108864),[org.tmplog,13] -> PartitionFetchInfo(0,67108864),[org.ecomlogin,0] -> PartitionFetchInfo(2,67108864),[org.payrequest,6] -> PartitionFetchInfo(6207,67108864),[org.dealrank,14] -> PartitionFetchInfo(3699,67108864),[org.wm_submitorder,15] -> PartitionFetchInfo(0,67108864),[org.web_seckill_ticketlog,2] -> PartitionFetchInfo(0,67108864),[org.nginx,4] -> PartitionFetchInfo(3122673,67108864),[org.mobile_group_push,11] -> PartitionFetchInfo(0,67108864),[org.data_resys_routerreq,11] -> PartitionFetchInfo(162226,67108864),[binlog.coupon,10] -> PartitionFetchInfo(32224,67108864),[log.eventlog,10] -> PartitionFetchInfo(31378,67108864),[log.intranet_nginx,2] -> PartitionFetchInfo(0,67108864),[org.extendapply,13] -> PartitionFetchInfo(1,67108864),[log.accesslog,0] -> PartitionFetchInfo(149156,67108864),[org.mobile,4] -> PartitionFetchInfo(51119,67108864),[org.ecomlogin,14] -> PartitionFetchInfo(655,67108864),[org.mobileapp,4] -> PartitionFetchInfo(150715,67108864),[org.mobile_push_netevent,14] -> PartitionFetchInfo(12863099,67108864),[org.wasp_card_log,4] -> PartitionFetchInfo(11,67108864),[org.data_resys_router,4] -> PartitionFetchInfo(35192,67108864),[org.nginx,15] -> PartitionFetchInfo(92880,67108864),[org.mobile_groupapi_search,6] -> PartitionFetchInfo(408,67108864),[org.wasp_page_visit_record,0] -> PartitionFetchInfo(0,67108864),
[log.couponverify,4] -> PartitionFetchInfo(0,67108864),[org.nginx,9] -> PartitionFetchInfo(100270,67108864),[org.mtcrm,7] -> PartitionFetchInfo(5078,67108864),[org.data_resys_hotquery,6] -> PartitionFetchInfo(83107,67108864),[org.mobile_nginx,6] -> PartitionFetchInfo(157869,67108864),[org.mobile,9] -> PartitionFetchInfo(1831700,67108864),[org.intranet_nginx,9] -> PartitionFetchInfo(132779,67108864),[org.mobile_push_applist,4] -> PartitionFetchInfo(2279570,67108864),[org.dealrank,0] -> PartitionFetchInfo(103,67108864),[org.mobile_networklog,3] -> PartitionFetchInfo(1723358,67108864),[org.mobile_movie_statistics,5] -> PartitionFetchInfo(6309,67108864),[org.mtrace,2] -> PartitionFetchInfo(94002,67108864),[org.data_asyncquerier_sql,7] -> PartitionFetchInfo(1,67108864),[org.spamlogin,12] -> PartitionFetchInfo(2480,67108864),[org.search_dealrank,8] -> PartitionFetchInfo(24118,67108864),[log.orderlog,1] -> PartitionFetchInfo(7231,67108864),[log.b,12] -> PartitionFetchInfo(0,67108864),[org.mobile_push_netevent,11] -> PartitionFetchInfo(659425,67108864),[org.mobile_networklog,9] -> PartitionFetchInfo(1251021,67108864),[org.data_resys_combinerec,11] -> PartitionFetchInfo(21870,67108864),[org.paytobiznotify,9] -> PartitionFetchInfo(0,67108864),[org.mobile_push_applist,12] -> PartitionFetchInfo(3099746,67108864),[org.mtrace,7] -> PartitionFetchInfo(1190072,67108864),[org.mobile_nginx,0] -> PartitionFetchInfo(174905,67108864),[org.web_user_message_op,8] -> PartitionFetchInfo(4529,67108864),[binlog.coupon,12] -> PartitionFetchInfo(23996,67108864),[org.tmplog,15] -> PartitionFetchInfo(0,67108864),[org.paylog,1] -> PartitionFetchInfo(489,67108864),[org.data_resys_routerres,7] -> PartitionFetchInfo(10502,67108864),[binlog.coupon,0] -> PartitionFetchInfo(25215,67108864),[org.nginxerrorlog,9] -> PartitionFetchInfo(29483,67108864),[org.mobile_push_applist,11] -> PartitionFetchInfo(103347,67108864),[org.basync,7] -> PartitionFetchInfo(299174,67108864),[org.mobile_push_netevent,0] -> PartitionFetchInfo(12720799,67108864),[org.basync,5] -> PartitionFetchInfo(4184,67108864),[org.mobile_nginx,12] -> PartitionFetchInfo(181939,67108864),[org.cos_errorlog,0] -> PartitionFetchInfo(189,67108864),[org.nginxerrorlog,15] -> PartitionFetchInfo(27555,67108864),[org.captcha,4] -> PartitionFetchInfo(14233,67108864),[org.data_search_smartbox,11] -> PartitionFetchInfo(0,67108864),[org.mobile_push_applist,2] -> PartitionFetchInfo(2371719,67108864),[org.mobile,5] -> PartitionFetchInfo(1906756,67108864),[org.mobile_movielog,3] -> PartitionFetchInfo(126780,67108864),[org.mobile_push_netevent,8] -> PartitionFetchInfo(10731244,67108864),[org.nginxerrorlog,14] -> PartitionFetchInfo(1525,67108864),[org.bdecomaction,2] -> PartitionFetchInfo(0,67108864),[org.cos_errorlog,12] -> PartitionFetchInfo(123,67108864),[org.mobile_networklog,2] -> PartitionFetchInfo(62332,67108864),[org.mobile_xmlog,8] -> PartitionFetchInfo(478,67108864),[org.mobile_eventlog,15] -> PartitionFetchInfo(8429,67108864),[org.cos_errorlog,10] -> PartitionFetchInfo(2569,67108864),[org.eventlog,4] -> PartitionFetchInfo(935,67108864),[org.data_asyncquerier_sql,14] -> PartitionFetchInfo(0,67108864),[org.dealrank,12] -> PartitionFetchInfo(93,67108864),[org.search_queryserver,1] -> PartitionFetchInfo(0,67108864),[binlog.user,5] -> PartitionFetchInfo(5950,67108864),[org.mobile_group_push,9] -> PartitionFetchInfo(25002,67108864),[org.mobile,13] -> PartitionFetchInfo(2117661,67108864),[org.nginx,11] -> PartitionFetchInfo(140093,67108864),[org.mobile_push_applist,7] -> PartitionFetchInfo(171384,67108864),[org.mobile_push_netevent,13] -> PartitionFetchInfo(531815,67108864),[org.mobile_eventlog,3] -> PartitionFetchInfo(14430,67108864),[org.captcha,6] -> PartitionFetchInfo(88,67108864),[org.mobile_push_netevent,5] -> PartitionFetchInfo(460960,67108864),[org.mobile_order_nomsg,4] -> PartitionFetchInfo(3,67108864),[org.data_xmltableview_access,2] -> PartitionFetchInfo(7,67108864),[org.payrequest,13] -> PartitionFetchInfo(101,67108864),[org.mtrace,8] -> Partiti
...
4),[org.mobile_mars_report,10] -> PartitionFetchInfo(2926,67108864),[org.mobile,12] -> PartitionFetchInfo(55550,67108864),[org.mobile_movielog,2] -> PartitionFetchInfo(1519,67108864),[org.commonsubscribe,3] -> PartitionFetchInfo(0,67108864),[org.apacheerrorlog,14] -> PartitionFetchInfo(1366,67108864),[org.wm_submitorder,13] -> PartitionFetchInfo(3,67108864),[org.data_resys_routerreq,9] -> PartitionFetchInfo(1582,67108864),[org.mobile_push_applist,1] -> PartitionFetchInfo(228829,67108864),[org.mtrace,11] -> PartitionFetchInfo(1232471,67108864),[org.404log,12] -> PartitionFetchInfo(0,67108864),[org.mtsg,4] -> PartitionFetchInfo(4,67108864),[org.applog,4] -> PartitionFetchInfo(607,67108864),[org.mobile_movielog,8] -> PartitionFetchInfo(3223,67108864),[org.mobile_push_netevent,6] -> PartitionFetchInfo(13081128,67108864),[org.cmsdealapi,11] -> PartitionFetchInfo(0,67108864),[org.mobile,2] -> PartitionFetchInfo(53029,67108864),[org.mobilerebindinfo,6] -> PartitionFetchInfo(32,67108864),[org.nginx,3] -> PartitionFetchInfo(105261,67108864),[org.mobile_push_applist,3] -> PartitionFetchInfo(162462,67108864),[org.mobile_xmlog,10] -> PartitionFetchInfo(21754,67108864),[org.mobile,6] -> PartitionFetchInfo(65015,67108864),[org.mobile_movielog,9] -> PartitionFetchInfo(132672,67108864),[org.intranet_nginx,11] -> PartitionFetchInfo(4738,67108864),[binlog.user,0] -> PartitionFetchInfo(225872,67108864) (kafka.server.ReplicaFetcherThread)
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:33)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:83)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:81)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.api.TopicData$.readFrom(FetchResponse.scala:81)
at kafka.api.FetchResponse$$anonfun$3.apply(FetchResponse.scala:142)
at kafka.api.FetchResponse$$anonfun$3.apply(FetchResponse.scala:141)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:141)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:112)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---------------------------------------------------------------------------
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala 2.9.2
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: newbie
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the buffer.capacity() gives 155733313, for some reason the buffer is smaller than the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if compression is used or not.
--
This message was sent by Atlassian JIRA
(v6.2#6252)