[jira] [Created] (TEZ-4567) Failed to load Lz4 codec

["Bilwa S T (Jira)" <ji...@apache.org>]

Bilwa S T created TEZ-4567:
------------------------------

             Summary: Failed to load Lz4 codec
                 Key: TEZ-4567
                 URL: https://issues.apache.org/jira/browse/TEZ-4567
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.10.3
            Reporter: Bilwa S T
            Assignee: Bilwa S T


Currently we use hadoop 3.3.6 version of hadoop. As part of this Jira HADOOP-17292, Lz4 is a provided dependency in Hadoop Common 3.3.1 for Lz4Codec, so we need to add the dependency in tez as well. Otherwise we get the below exception when we run hive job on tez
{code:java}
Caused by: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66) at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152) at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131) at org.apache.hadoop.io.compress.Lz4Codec.createOutputStream(Lz4Codec.java:70) at org.apache.hadoop.hive.ql.exec.Utilities.createCompressedStream(Utilities.java:949) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922) at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133) at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45) at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110) at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64) at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116) at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:556) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92) ... 19 more Caused by: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Factory at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 53 more{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)