You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vinay Gupta <vi...@yahoo.com.INVALID> on 2014/06/06 06:29:46 UTC
Trouble with snappy and SimpleConsumer
Hi,
I am using
kafka_2.9.2-0.8.1 and
snappy-java-1.1.0.1.jar
I have been able to successfully use gzip with the same library. however “snappy” doesn’t work in consumer side.
Producer is able to send snappy messages to broker though.
I have made sure that snappy java lib is the same on both consumer and producer side.
Is there a problem with 1.1.0.1 snappy for usage with kafka ??? Should I switch to an older version??
Anyone else having this problem?? Thanks in advance for any pointers.
Thanks
-Vinay
——————————Exception ——————————————
topic=events partition=2leaderId=3 beginOffset=2225451 offset=2225451 checksum=0 time=1402008438921
java.lang.Exception: org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
at java.io.InputStream.read(InputStream.java:82)
at kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply$mcI$sp(Unknown Source)
at kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(Unknown Source)
at kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(Unknown Source)
at scala.collection.immutable.Stream$.continually(Stream.scala:1129)
at kafka.message.ByteBufferMessageSet$.decompress(Unknown Source)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(Unknown Source)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(Unknown Source)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(Unknown Source)
at kafka.utils.IteratorTemplate.maybeComputeNext(Unknown Source)
at kafka.utils.IteratorTemplate.hasNext(Unknown Source)
at kafka.javaapi.message.ByteBufferMessageSet$$anon$1.hasNext(Unknown Source)
:
:
:
:
Caused by: java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
... 31 more
Re: Trouble with snappy and SimpleConsumer
Posted by Chi Hoang <ch...@groupon.com>.
We are using Kafka 0.8.1.1 (2.9.2) rebuilt with Snappy 1.1.0.1, and don't
have problems with producers or consumers.
Chi
On Thu, Jun 5, 2014 at 9:29 PM, Vinay Gupta <vi...@yahoo.com.invalid>
wrote:
> Hi,
> I am using
> kafka_2.9.2-0.8.1 and
> snappy-java-1.1.0.1.jar
>
>
> I have been able to successfully use gzip with the same library. however
> “snappy” doesn’t work in consumer side.
> Producer is able to send snappy messages to broker though.
> I have made sure that snappy java lib is the same on both consumer and
> producer side.
>
> Is there a problem with 1.1.0.1 snappy for usage with kafka ??? Should I
> switch to an older version??
>
> Anyone else having this problem?? Thanks in advance for any pointers.
>
>
> Thanks
> -Vinay
>
>
> ——————————Exception ——————————————
>
> topic=events partition=2leaderId=3 beginOffset=2225451 offset=2225451
> checksum=0 time=1402008438921
> java.lang.Exception:
> org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
> at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
> at
> org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
> at
> org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
> at
> org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
> at java.io.InputStream.read(InputStream.java:82)
> at
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply$mcI$sp(Unknown
> Source)
> at
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(Unknown
> Source)
> at
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(Unknown
> Source)
> at
> scala.collection.immutable.Stream$.continually(Stream.scala:1129)
> at kafka.message.ByteBufferMessageSet$.decompress(Unknown Source)
> at
> kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(Unknown Source)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(Unknown
> Source)
> at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(Unknown
> Source)
> at kafka.utils.IteratorTemplate.maybeComputeNext(Unknown Source)
> at kafka.utils.IteratorTemplate.hasNext(Unknown Source)
> at
> kafka.javaapi.message.ByteBufferMessageSet$$anon$1.hasNext(Unknown Source)
> :
> :
> :
> :
> Caused by: java.lang.UnsatisfiedLinkError:
> org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
> ... 31 more
>
>
>