You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Siegel <ds...@knewton.com> on 2012/07/10 06:02:07 UTC
InvalidMessageSizeException: null
I've just gotten the following error while running the zookeeper consumer.
I made a backup of the kafka log directory and wiped the logs. I
restarting kafka and the consumer. After processing a few hundred
messages successfully I got the same error again. I restarted the consumer
again and got the same error immediately.
I'm running Kafka 0.7.1
I've included a sample of DumpLogSegments. The rest of the dumps looked
the same.
Thanks for your help.
-David Siegel
2012-07-10 02:31:21,998 ERROR [Consumer1]
c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
kafka.common.InvalidMessageSizeException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[na:1.6.0_30]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
~[na:1.6.0_30]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
~[na:1.6.0_30]
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
~[na:1.6.0_30]
at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
at
kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
~[KPIP-0.4.birdy.jar:na]
at
kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
~[KPIP-0.4.birdy.jar:na]
at
kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
~[KPIP-0.4.birdy.jar:na]
at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81)
~[KPIP-0.4.birdy.jar:na]
at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
~[KPIP-0.4.birdy.jar:na]
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
~[KPIP-0.4.birdy.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
2012-07-10 02:31:21,998 ERROR [Consumer1]
c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread
exiting
java.lang.IllegalStateException: Iterator is in failed state
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47)
~[KPIP-0.4.birdy.jar:na]
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
~[KPIP-0.4.birdy.jar:na]
at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
~[KPIP-0.4.birdy.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka
Starting offset: 0
offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec:
NoCompressionCodec
offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec:
NoCompressionCodec
offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec:
NoCompressionCodec
offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec:
NoCompressionCodec
offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec:
NoCompressionCodec
offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec:
NoCompressionCodec
offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec:
NoCompressionCodec
offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec:
NoCompressionCodec
offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec:
NoCompressionCodec
offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec:
NoCompressionCodec
offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec:
NoCompressionCodec
offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec:
NoCompressionCodec
offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec:
NoCompressionCodec
offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec:
NoCompressionCodec
offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec:
NoCompressionCodec
tail of the log is at offset: 8955
Re: InvalidMessageSizeException: null
Posted by David Siegel <ds...@knewton.com>.
It is reproducible. I've created a ticket with the logs attached. Thanks
for your help.
https://issues.apache.org/jira/browse/KAFKA-397
d
On Tue, Jul 10, 2012 at 12:53 PM, Neha Narkhede <ne...@gmail.com>wrote:
> David,
>
> I suspect this is a bug with the MultiFetchResponse. Is this reproducible ?
> Would you mind filing a bug with the log files for the topic attached ?
>
> Thanks,
> Neha
>
>
> On Tue, Jul 10, 2012 at 8:41 AM, David Siegel <ds...@knewton.com> wrote:
> > I've copied the relevant configuration options below. Doesn't this
> > exception normally print out a useful error instead of just "null"?
> >
> > producer config:
> > max.message.size: 1000000
> > consumer config:
> > fetch.size: 2072000
> >
> > On Tue, Jul 10, 2012 at 10:57 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> David,
> >>
> >> The fetch request gets an InvalidMessageSizeException. This means either
> >> log corruption or the fetch size is smaller than the largest message.
> Could
> >> you check your fetch size?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Mon, Jul 9, 2012 at 9:02 PM, David Siegel <ds...@knewton.com>
> wrote:
> >>
> >> > I've just gotten the following error while running the zookeeper
> >> consumer.
> >> >
> >> > I made a backup of the kafka log directory and wiped the logs. I
> >> > restarting kafka and the consumer. After processing a few hundred
> >> > messages successfully I got the same error again. I restarted the
> >> consumer
> >> > again and got the same error immediately.
> >> >
> >> > I'm running Kafka 0.7.1
> >> >
> >> > I've included a sample of DumpLogSegments. The rest of the dumps
> looked
> >> > the same.
> >> >
> >> > Thanks for your help.
> >> >
> >> > -David Siegel
> >> >
> >> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
> >> > c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> >> > kafka.common.InvalidMessageSizeException: null
> >> > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> > Method) ~[na:1.6.0_30]
> >> > at
> >> >
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> > ~[na:1.6.0_30]
> >> > at
> >> >
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> > ~[na:1.6.0_30]
> >> > at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> > ~[na:1.6.0_30]
> >> > at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> >> > at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> >> > at
> >> > kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> >
> >> >
> >>
> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> >
> >>
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> >
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> >> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
> >> > c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state.
> >> Thread
> >> > exiting
> >> > java.lang.IllegalStateException: Iterator is in failed state
> >> > at
> >> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at
> >> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> >> > ~[KPIP-0.4.birdy.jar:na]
> >> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> >> >
> >> > Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka
> >> > Starting offset: 0
> >> > offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> >> > NoCompressionCodec
> >> > tail of the log is at offset: 8955
> >> >
> >>
>
Re: InvalidMessageSizeException: null
Posted by Neha Narkhede <ne...@gmail.com>.
David,
I suspect this is a bug with the MultiFetchResponse. Is this reproducible ?
Would you mind filing a bug with the log files for the topic attached ?
Thanks,
Neha
On Tue, Jul 10, 2012 at 8:41 AM, David Siegel <ds...@knewton.com> wrote:
> I've copied the relevant configuration options below. Doesn't this
> exception normally print out a useful error instead of just "null"?
>
> producer config:
> max.message.size: 1000000
> consumer config:
> fetch.size: 2072000
>
> On Tue, Jul 10, 2012 at 10:57 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> David,
>>
>> The fetch request gets an InvalidMessageSizeException. This means either
>> log corruption or the fetch size is smaller than the largest message. Could
>> you check your fetch size?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Jul 9, 2012 at 9:02 PM, David Siegel <ds...@knewton.com> wrote:
>>
>> > I've just gotten the following error while running the zookeeper
>> consumer.
>> >
>> > I made a backup of the kafka log directory and wiped the logs. I
>> > restarting kafka and the consumer. After processing a few hundred
>> > messages successfully I got the same error again. I restarted the
>> consumer
>> > again and got the same error immediately.
>> >
>> > I'm running Kafka 0.7.1
>> >
>> > I've included a sample of DumpLogSegments. The rest of the dumps looked
>> > the same.
>> >
>> > Thanks for your help.
>> >
>> > -David Siegel
>> >
>> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
>> > c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
>> > kafka.common.InvalidMessageSizeException: null
>> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method) ~[na:1.6.0_30]
>> > at
>> >
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> > ~[na:1.6.0_30]
>> > at
>> >
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> > ~[na:1.6.0_30]
>> > at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> > ~[na:1.6.0_30]
>> > at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
>> > at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
>> > at
>> > kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> >
>> >
>> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> >
>> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
>> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
>> > c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state.
>> Thread
>> > exiting
>> > java.lang.IllegalStateException: Iterator is in failed state
>> > at
>> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at
>> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
>> > ~[KPIP-0.4.birdy.jar:na]
>> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
>> >
>> > Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka
>> > Starting offset: 0
>> > offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec:
>> > NoCompressionCodec
>> > tail of the log is at offset: 8955
>> >
>>
Re: InvalidMessageSizeException: null
Posted by David Siegel <ds...@knewton.com>.
I've copied the relevant configuration options below. Doesn't this
exception normally print out a useful error instead of just "null"?
producer config:
max.message.size: 1000000
consumer config:
fetch.size: 2072000
On Tue, Jul 10, 2012 at 10:57 AM, Jun Rao <ju...@gmail.com> wrote:
> David,
>
> The fetch request gets an InvalidMessageSizeException. This means either
> log corruption or the fetch size is smaller than the largest message. Could
> you check your fetch size?
>
> Thanks,
>
> Jun
>
> On Mon, Jul 9, 2012 at 9:02 PM, David Siegel <ds...@knewton.com> wrote:
>
> > I've just gotten the following error while running the zookeeper
> consumer.
> >
> > I made a backup of the kafka log directory and wiped the logs. I
> > restarting kafka and the consumer. After processing a few hundred
> > messages successfully I got the same error again. I restarted the
> consumer
> > again and got the same error immediately.
> >
> > I'm running Kafka 0.7.1
> >
> > I've included a sample of DumpLogSegments. The rest of the dumps looked
> > the same.
> >
> > Thanks for your help.
> >
> > -David Siegel
> >
> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
> > c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> > kafka.common.InvalidMessageSizeException: null
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method) ~[na:1.6.0_30]
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > ~[na:1.6.0_30]
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > ~[na:1.6.0_30]
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > ~[na:1.6.0_30]
> > at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> > at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> > at
> > kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> >
> >
> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> >
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
> > ~[KPIP-0.4.birdy.jar:na]
> > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> > ~[KPIP-0.4.birdy.jar:na]
> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> > 2012-07-10 02:31:21,998 ERROR [Consumer1]
> > c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state.
> Thread
> > exiting
> > java.lang.IllegalStateException: Iterator is in failed state
> > at
> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47)
> > ~[KPIP-0.4.birdy.jar:na]
> > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> > ~[KPIP-0.4.birdy.jar:na]
> > at
> kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> > ~[KPIP-0.4.birdy.jar:na]
> > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> >
> > Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka
> > Starting offset: 0
> > offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> > NoCompressionCodec
> > offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> > NoCompressionCodec
> > tail of the log is at offset: 8955
> >
>
Re: InvalidMessageSizeException: null
Posted by Jun Rao <ju...@gmail.com>.
David,
The fetch request gets an InvalidMessageSizeException. This means either
log corruption or the fetch size is smaller than the largest message. Could
you check your fetch size?
Thanks,
Jun
On Mon, Jul 9, 2012 at 9:02 PM, David Siegel <ds...@knewton.com> wrote:
> I've just gotten the following error while running the zookeeper consumer.
>
> I made a backup of the kafka log directory and wiped the logs. I
> restarting kafka and the consumer. After processing a few hundred
> messages successfully I got the same error again. I restarted the consumer
> again and got the same error immediately.
>
> I'm running Kafka 0.7.1
>
> I've included a sample of DumpLogSegments. The rest of the dumps looked
> the same.
>
> Thanks for your help.
>
> -David Siegel
>
> 2012-07-10 02:31:21,998 ERROR [Consumer1]
> c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.6.0_30]
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> ~[na:1.6.0_30]
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at
> kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
> ~[KPIP-0.4.birdy.jar:na]
> at
>
> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
> ~[KPIP-0.4.birdy.jar:na]
> at
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
> ~[KPIP-0.4.birdy.jar:na]
> at
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81)
> ~[KPIP-0.4.birdy.jar:na]
> at
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> ~[KPIP-0.4.birdy.jar:na]
> at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1]
> c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. Thread
> exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47)
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36)
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43)
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
>
> Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka
> Starting offset: 0
> offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec:
> NoCompressionCodec
> offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec:
> NoCompressionCodec
> tail of the log is at offset: 8955
>