You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "John Fung (JIRA)" <ji...@apache.org> on 2012/09/24 23:50:07 UTC
[jira] [Created] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
John Fung created KAFKA-528:
-------------------------------
Summary: IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
Key: KAFKA-528
URL: https://issues.apache.org/jira/browse/KAFKA-528
Project: Kafka
Issue Type: Bug
Reporter: John Fung
1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
2. The system test output log can be found at: system_test_1348521165/system_test_output.log
3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
[2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:512)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
at kafka.message.Message.magic(Message.scala:119)
at kafka.message.Message.checksum(Message.scala:132)
at kafka.message.Message.isValid(Message.scala:144)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-528) IndexOutOfBoundsException
thrown by kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466086#comment-13466086 ]
Jun Rao edited comment on KAFKA-528 at 9/29/12 1:14 PM:
--------------------------------------------------------
Attach patch v2. Fixed one additional bug in KafkaController.
The bug is that Map(("a",0)->0,("a",1)->0)).map(_,_1) returns Map("a" -> 1) while we expect it to be Seq(("a,0),("a",1)). Fixed it by using .keySet instead.
With patch v2, our system test now passes.
was (Author: junrao):
Attach patch v2. Fixed one additional bug in KafkaController.
The bug is that Map(("a",0)->0,("a",1)->)).map(_,_1) returns Map("a" -> 1) while we expect it to be Seq(("a,0),("a",1)). Fixed it by using .keySet instead.
With patch v2, our system test now passes.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, kafka-528_v2.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reassigned KAFKA-528:
-----------------------------
Assignee: Jun Rao
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-528:
--------------------------
Attachment: kafka-528_v2.patch
Attach patch v2. Fixed one additional bug in KafkaController.
The bug is that Map(("a",0)->0,("a",1)->)).map(_,_1) returns Map("a" -> 1) while we expect it to be Seq(("a,0),("a",1)). Fixed it by using .keySet instead.
With patch v2, our system test now passes.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, kafka-528_v2.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-528:
--------------------------
Fix Version/s: 0.8
Labels: bugs (was: )
Affects Version/s: 0.8
Status: Patch Available (was: Open)
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-528.
-------------------------
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, kafka-528_v2.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466271#comment-13466271 ]
Neha Narkhede commented on KAFKA-528:
-------------------------------------
+1 on patch v2.
Minor comments on logging -
1. The debug statement in Replica says "Updated highwatermark to ". It will be good to know the topic and partition as well.
2. The println in DumpLogSegment says "Log is corrupted from... Is corrupted". Let's fix that.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, kafka-528_v2.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-528:
--------------------------
Priority: Blocker (was: Major)
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466002#comment-13466002 ]
John Fung commented on KAFKA-528:
---------------------------------
Tested kafka-528.patch with rev. 1389460 and there is no data loss now. However, there is still another exception showing in broker log (already filed under KAFKA-529) :
[2012-09-28 23:05:54,669] ERROR [ReplicaFetcherThread-2-0-on-broker-3], Error due to (kafka.server.ReplicaFetcherThread)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:514)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
at kafka.message.Message.magic(Message.scala:119)
at kafka.message.Message.checksum(Message.scala:132)
at kafka.message.Message.isValid(Message.scala:144)
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-528:
----------------------------
Attachment: system_test_1348521165.tar.gz
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Reporter: John Fung
> Attachments: system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-528:
--------------------------
Attachment: kafka-528.patch
Attach a patch. There are a couple of critical issues.
1. FileMessageSet.truncateTo(): There is a java bug in FileChannel.truncate. The problem is that file size is set correctly, but file position is not changed accordingly. Added a fix that manually sets the file position.
2. ZookeeperConsumerConnector: The call to get the initial offset sometimes fails since it's connected to a broker that's not the leader of a partition. Changed it to set the initial offset in ConsumerFetcherThread.
Also fixed a few other issues:
3. Partition: There is unnecessary ISR expansion.
4. SyncProducer: lastConnectionTime should be set in connect().
5. DumpLogSegment: Report if log is corrupted at the end of a file.
6. Added some more tracing code.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-528:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the review. Committed to 0.8 with some minor changes.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, kafka-528_v2.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-528) IndexOutOfBoundsException thrown by
kafka.consumer.ConsumerFetcherThread
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465852#comment-13465852 ]
Jay Kreps commented on KAFKA-528:
---------------------------------
+1 Lgtm.
You made the logging for leaderAndIsr request to be info level. If it is info level it would be good to rephrase that in a way folks would be able to understand, otherwise we have a log full of really esoteric things that only make sense to us.
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-528) IndexOutOfBoundsException
thrown by kafka.consumer.ConsumerFetcherThread
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466002#comment-13466002 ]
John Fung edited comment on KAFKA-528 at 9/29/12 10:21 AM:
-----------------------------------------------------------
1. Tested kafka-528.patch with rev. 1389460 and there is no data loss now. The Exception specified in this JIRA is not showing.
2. However, there is still another exception showing in broker log (already filed under KAFKA-529) :
[2012-09-28 23:05:54,669] ERROR [ReplicaFetcherThread-2-0-on-broker-3], Error due to (kafka.server.ReplicaFetcherThread)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:514)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
at kafka.message.Message.magic(Message.scala:119)
at kafka.message.Message.checksum(Message.scala:132)
at kafka.message.Message.isValid(Message.scala:144)
was (Author: jfung):
Tested kafka-528.patch with rev. 1389460 and there is no data loss now. However, there is still another exception showing in broker log (already filed under KAFKA-529) :
[2012-09-28 23:05:54,669] ERROR [ReplicaFetcherThread-2-0-on-broker-3], Error due to (kafka.server.ReplicaFetcherThread)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:514)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
at kafka.message.Message.magic(Message.scala:119)
at kafka.message.Message.checksum(Message.scala:132)
at kafka.message.Message.isValid(Message.scala:144)
> IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
> ------------------------------------------------------------------------
>
> Key: KAFKA-528
> URL: https://issues.apache.org/jira/browse/KAFKA-528
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: John Fung
> Assignee: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: kafka-528.patch, system_test_1348521165.tar.gz
>
>
> 1. Attached file system_test_1348521165.tar.gz contains all the associated log files for this test session.
> 2. The system test output log can be found at: system_test_1348521165/system_test_output.log
> 3. The following log message can be found at: system_test_1348521165/logs/console_consumer-5/console_consumer.log
> [2012-09-24 14:15:12,016] ERROR [ConsumerFetcherThread-console-consumer-16186_jfung-1348521311426-2c83ced7-0-1], Error due to (kafka.consumer.ConsumerFetcherThread)
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:121)
> at kafka.message.Message.magic(Message.scala:119)
> at kafka.message.Message.checksum(Message.scala:132)
> at kafka.message.Message.isValid(Message.scala:144)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:118)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:149)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:89)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:53)
> at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:54)
> at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:54)
> at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:103)
> at kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:96)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:127)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:96)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira