You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Xuyen On <xo...@ancestry.com> on 2014/05/13 01:08:48 UTC

IndexOutOfBoundsException error

Hi all,

I am seeing some corrupt data on my 0.7.2 Kafka cluster. Every once in awhile I'll get the following message:

Exception in thread "kafka-consumer" java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Unknown Source)
        at java.nio.HeapByteBuffer.get(Unknown Source)
        at kafka.message.Message.magic(Message.scala:133)
        at kafka.message.Message.checksum(Message.scala:146)
        at kafka.message.Message.isValid(Message.scala:158)
        at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:129)
        at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:166)
        at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100)
        at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
        at scala.collection.Iterator$class.foreach(Iterator.scala:631)
        at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
        at kafka.message.MessageSet.foreach(MessageSet.scala:87)
        at kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:97)
        at kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:94)
        at scala.collection.Iterator$class.foreach(Iterator.scala:631)
        at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
        at kafka.api.MultiFetchResponse.foreach(MultiFetchResponse.scala:25)
        at kafka.tools.SimpleConsumerShell$$anon$1.run(SimpleConsumerShell.scala:94)

The only way I can get past this error is to move the offset past the current message to skip the bad data. I use the simpleConsumer.getOffsetsBefore() method to get a valid lists of offsets and I skip to the next valid offset. 

Has anyone encountered this issue before? I found a message where someone experienced similar issues and they were running JDK 1.7 and was able to get around it by running JDK 1.6. I am running OpenJDK 1.7 but I tried running Sun JDK 1.6 but I still have the same problem.

My guess is that the data is corrupt somehow but I don't know how to look at the raw data to confirm. Can anyone suggest ideas for me to debug this issue?

Thanks,

Xuyen

RE: IndexOutOfBoundsException error

Posted by Xuyen On <xo...@ancestry.com>.

We decided to clear the old logs and start fresh. I'll run that app the next time if we run into the same problem again.

Thanks Jun,

Xuyen

-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Monday, May 12, 2014 8:58 PM
To: users@kafka.apache.org
Subject: Re: IndexOutOfBoundsException error

Could you run the DumpLogSegments tool to see if there are indeed corrupted messages in the broker log?

Thanks,

Jun


On Mon, May 12, 2014 at 4:08 PM, Xuyen On <xo...@ancestry.com> wrote:

> Hi all,
>
> I am seeing some corrupt data on my 0.7.2 Kafka cluster. Every once in 
> awhile I'll get the following message:
>
> Exception in thread "kafka-consumer" java.lang.IndexOutOfBoundsException
>         at java.nio.Buffer.checkIndex(Unknown Source)
>         at java.nio.HeapByteBuffer.get(Unknown Source)
>         at kafka.message.Message.magic(Message.scala:133)
>         at kafka.message.Message.checksum(Message.scala:146)
>         at kafka.message.Message.isValid(Message.scala:158)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:129)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:166)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100)
>         at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
>         at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>         at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
>         at kafka.message.MessageSet.foreach(MessageSet.scala:87)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:97)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:94)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>         at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
>         at
> kafka.api.MultiFetchResponse.foreach(MultiFetchResponse.scala:25)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1.run(SimpleConsumerShell.scala:
> 94)
>
> The only way I can get past this error is to move the offset past the 
> current message to skip the bad data. I use the
> simpleConsumer.getOffsetsBefore() method to get a valid lists of 
> offsets and I skip to the next valid offset.
>
> Has anyone encountered this issue before? I found a message where 
> someone experienced similar issues and they were running JDK 1.7 and 
> was able to get around it by running JDK 1.6. I am running OpenJDK 1.7 
> but I tried running Sun JDK 1.6 but I still have the same problem.
>
> My guess is that the data is corrupt somehow but I don't know how to 
> look at the raw data to confirm. Can anyone suggest ideas for me to 
> debug this issue?
>
> Thanks,
>
> Xuyen
>
>
>

Re: IndexOutOfBoundsException error

Posted by Jun Rao <ju...@gmail.com>.

Could you run the DumpLogSegments tool to see if there are indeed corrupted
messages in the broker log?

Thanks,

Jun


On Mon, May 12, 2014 at 4:08 PM, Xuyen On <xo...@ancestry.com> wrote:

> Hi all,
>
> I am seeing some corrupt data on my 0.7.2 Kafka cluster. Every once in
> awhile I'll get the following message:
>
> Exception in thread "kafka-consumer" java.lang.IndexOutOfBoundsException
>         at java.nio.Buffer.checkIndex(Unknown Source)
>         at java.nio.HeapByteBuffer.get(Unknown Source)
>         at kafka.message.Message.magic(Message.scala:133)
>         at kafka.message.Message.checksum(Message.scala:146)
>         at kafka.message.Message.isValid(Message.scala:158)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:129)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:166)
>         at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100)
>         at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
>         at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>         at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
>         at kafka.message.MessageSet.foreach(MessageSet.scala:87)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:97)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:94)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:631)
>         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>         at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
>         at
> kafka.api.MultiFetchResponse.foreach(MultiFetchResponse.scala:25)
>         at
> kafka.tools.SimpleConsumerShell$$anon$1.run(SimpleConsumerShell.scala:94)
>
> The only way I can get past this error is to move the offset past the
> current message to skip the bad data. I use the
> simpleConsumer.getOffsetsBefore() method to get a valid lists of offsets
> and I skip to the next valid offset.
>
> Has anyone encountered this issue before? I found a message where someone
> experienced similar issues and they were running JDK 1.7 and was able to
> get around it by running JDK 1.6. I am running OpenJDK 1.7 but I tried
> running Sun JDK 1.6 but I still have the same problem.
>
> My guess is that the data is corrupt somehow but I don't know how to look
> at the raw data to confirm. Can anyone suggest ideas for me to debug this
> issue?
>
> Thanks,
>
> Xuyen
>
>
>