You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "David Siegel (JIRA)" <ji...@apache.org> on 2012/07/10 19:22:33 UTC
[jira] [Created] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
David Siegel created KAFKA-397:
----------------------------------
Summary: kafka.common.InvalidMessageSizeException: null
Key: KAFKA-397
URL: https://issues.apache.org/jira/browse/KAFKA-397
Project: Kafka
Issue Type: Bug
Affects Versions: 0.7.1
Reporter: David Siegel
Priority: Blocker
I've just gotten the following error while running the zookeeper consumer.
I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
I'm running Kafka 0.7.1
I have the following message size configurations:
producer config:
max.message.size: 1000000
consumer config:
fetch.size: 2072000
Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
kafka.common.InvalidMessageSizeException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
java.lang.IllegalStateException: Iterator is in failed state
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
Posted by "David Siegel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Siegel updated KAFKA-397:
-------------------------------
Attachment: kafka-logs-invalid-message-size-null.tar
> kafka.common.InvalidMessageSizeException: null
> ----------------------------------------------
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: David Siegel
> Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.
> I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
Posted by "David Siegel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410660#comment-13410660 ]
David Siegel commented on KAFKA-397:
------------------------------------
When we set the number of partitions per topic to 100 and set the consumer to use 100 threads, we don't see this error.
When we set the number of partitions per topic to 100 and set the consumers to use 2 threads, we see the error consistently.
> kafka.common.InvalidMessageSizeException: null
> ----------------------------------------------
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: David Siegel
> Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.
> I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410671#comment-13410671 ]
Jun Rao commented on KAFKA-397:
-------------------------------
Yes, if you wipe off the log. You need to clean up offsets in ZK too.
Could you also check if the broker has any error log with the following text?
"error when processing request "
> kafka.common.InvalidMessageSizeException: null
> ----------------------------------------------
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: David Siegel
> Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.
> I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
Posted by "David Arthur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486297#comment-13486297 ]
David Arthur commented on KAFKA-397:
------------------------------------
Try cleaning out ZooKeeper as well as the Kafka logs (as Jun suggested). Based on your exception, I'd guess that your consumer is using the last offset stored in ZooKeeper which exceeds the current size of your logs. I have seen exceptions like this when screwing around with the log files.
> kafka.common.InvalidMessageSizeException: null
> ----------------------------------------------
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: David Siegel
> Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.
> I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-397)
kafka.common.InvalidMessageSizeException: null
Posted by "David Siegel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410691#comment-13410691 ]
David Siegel commented on KAFKA-397:
------------------------------------
2012-07-10T18:07:27.27635 [2012-07-10 18:07:27,275] ERROR error when processing request FetchRequest(topic:tmp, part:0 offset:2411 maxSize:2072000) (kafka.server
.KafkaRequestHandlers)
2012-07-10T18:07:27.27638 kafka.common.OffsetOutOfRangeException: offset 2411 is out of range
2012-07-10T18:07:27.27639 at kafka.log.Log$.findRange(Log.scala:46)
2012-07-10T18:07:27.27640 at kafka.log.Log.read(Log.scala:247)
2012-07-10T18:07:27.27640 at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:108)
2012-07-10T18:07:27.27641 at kafka.server.KafkaRequestHandlers$$anonfun$2.apply(KafkaRequestHandlers.scala:97)
2012-07-10T18:07:27.27641 at kafka.server.KafkaRequestHandlers$$anonfun$2.apply(KafkaRequestHandlers.scala:96)
2012-07-10T18:07:27.27642 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
2012-07-10T18:07:27.27642 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
2012-07-10T18:07:27.27643 at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
2012-07-10T18:07:27.27644 at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
2012-07-10T18:07:27.27644 at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
2012-07-10T18:07:27.27645 at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
2012-07-10T18:07:27.27646 at kafka.server.KafkaRequestHandlers.handleMultiFetchRequest(KafkaRequestHandlers.scala:96)
2012-07-10T18:07:27.27647 at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:40)
2012-07-10T18:07:27.27648 at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:40)
2012-07-10T18:07:27.27648 at kafka.network.Processor.handle(SocketServer.scala:296)
2012-07-10T18:07:27.27649 at kafka.network.Processor.read(SocketServer.scala:319)
2012-07-10T18:07:27.27649 at kafka.network.Processor.run(SocketServer.scala:214)
2012-07-10T18:07:27.27650 at java.lang.Thread.run(Thread.java:662)
> kafka.common.InvalidMessageSizeException: null
> ----------------------------------------------
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: David Siegel
> Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.
> I made a backup of the kafka log directory and wiped the logs. I restarting kafka and the consumer. After processing a few hundred messages successfully I got the same error again. I restarted the consumer again and got the same error immediately. I wiped the logs yet again and reproduced the error again. I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_30]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) ~[na:1.6.0_30]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) ~[KPIP-0.4.birdy.jar:na]
> at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira