[jira] [Reopened] (KAFKA-8622) Snappy Compression Not Working

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax reopened KAFKA-8622:
------------------------------------

> Snappy Compression Not Working
> ------------------------------
>
>                 Key: KAFKA-8622
>                 URL: https://issues.apache.org/jira/browse/KAFKA-8622
>             Project: Kafka
>          Issue Type: Bug
>          Components: compression
>    Affects Versions: 2.3.0, 2.2.1
>            Reporter: Kunal Verma
>            Assignee: kaushik srinivas
>            Priority: Major
>
> I am trying to produce a message on the broker with compression enabled as snappy.
> Environment :
> Brokers[Kafka-cluster] are hosted on Centos 7
> I have download the latest version (2.3.0 & 2.2.1) tar, extract it and moved to /opt/kafka-<version>
> I have executed the broker with standard configuration.
> In my producer service(written in java), I have enabled snappy compression.
> props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
>  
> so while sending record on broker, I am getting following errors:
> org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
>  
> While investing further at broker end I got following error in log
>  
> logs/kafkaServer.out:java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-ecd381af-ffdd-4a5c-a3d8-b802d0fa4e85-libsnappyjava.so: /tmp/snappy-1.1.7-ecd381af-ffdd-4a5c-a3d8-b802d0fa4e85-libsnappyjava.so: failed to map segment from shared object: Operation not permitted
> ------
>  
> [2019-07-02 15:29:43,399] ERROR [ReplicaManager broker=1] Error processing append operation on partition test-bulk-1 (kafka.server.ReplicaManager)
> java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
> at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435)
> at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:466)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:168)
> at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:293)
> at org.apache.kafka.common.record.DefaultRecordBatch$1.readNext(DefaultRecordBatch.java:264)
> at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:569)
> at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:538)
> at org.apache.kafka.common.record.DefaultRecordBatch.iterator(DefaultRecordBatch.java:327)
> at scala.collection.convert.Wrappers$JIterableWrapper.iterator(Wrappers.scala:55)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at kafka.log.LogValidator$.$anonfun$validateMessagesAndAssignOffsetsCompressed$1(LogValidator.scala:269)
> at kafka.log.LogValidator$.$anonfun$validateMessagesAndAssignOffsetsCompressed$1$adapted(LogValidator.scala:261)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(LogValidator.scala:261)
> at kafka.log.LogValidator$.validateMessagesAndAssignOffsets(LogValidator.scala:73)
> at kafka.log.Log.liftedTree1$1(Log.scala:881)
> at kafka.log.Log.$anonfun$append$2(Log.scala:868)
> at kafka.log.Log.maybeHandleIOException(Log.scala:2065)
> at kafka.log.Log.append(Log.scala:850)
> at kafka.log.Log.appendAsLeader(Log.scala:819)
> at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:772)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:259)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:759)
> at kafka.server.ReplicaManager.$anonfun$appendToLocalLog$2(ReplicaManager.scala:763)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
> at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
> at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
> at scala.collection.TraversableLike.map(TraversableLike.scala:237)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:751)
> at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:492)
> at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:544)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:113)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
> at java.lang.Thread.run(Thread.java:748)
>  
>  
> -------
> I have checked the snappy jar is on the classpath. Please refer
>  
> Client environment:java.class.path=/opt/kafka_2.12-2.3.0/bin/../libs/activation-1.1.1.jar:/opt/kafka_2.12-2.3.0/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-api-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-file-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-json-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/guava-20.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka_2.12-2.3.0/bin/..libs/hk2-utils-2.5.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jackson-module-scala_2.12-2.9.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka_2.12-2.3.0/bin/../lib/jakarta.inject-2.5.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka_2.12-2.3.0/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka_2.12-2.3.0/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-client-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-common-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jersey-server-2.28.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka_2.12-2.3.0/bin/../libs/jsr305-3.0.2.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0-sources.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-streams-scala_2.12-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/log4j-1.2.17.jar:/opt/kafka_2.12-2.3.0/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka_2.12-2.3.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka_2.12-2.3.0/bin/../libs/paranamer-2.8.jar:/opt/kafka_2.12-2.3.0/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/reflections-0.9.11.jar:/opt/kafka_2.12-2.3.0/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka_2.12-2.3.0/bin/../libs/scala-library-2.12.8.jar:/opt/kafka_2.12-2.3.0/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka_2.12-2.3.0/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka_2.12-2.3.0/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka_2.12-2.3.0/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka_2.12-2.3.0/bin/../libs/*snappy-java-1.1.7.3.jar*:/opt/kafka_2.12-2.3.0/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka_2.12-2.3.0/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka_2.12-2.3.0/bin/../libs/zkclient-0.11.jar:/opt/kafka_2.12-2.3.0/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka_2.12-2.3.0/bin/../libs/zstd-jni-1.4.0-1.jar
>  
>  
> Kindly suggest the solution. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)