You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Dmitriy Fingerman (Jira)" <ji...@apache.org> on 2023/02/24 19:23:00 UTC
[jira] [Created] (HIVE-27105) Querying parquet table with zstd encryption is failing
Dmitriy Fingerman created HIVE-27105:
----------------------------------------
Summary: Querying parquet table with zstd encryption is failing
Key: HIVE-27105
URL: https://issues.apache.org/jira/browse/HIVE-27105
Project: Hive
Issue Type: Bug
Reporter: Dmitriy Fingerman
Assignee: Dmitriy Fingerman
Steps to reproduce on the local with upstream or downstream code:
* Start HiveServer2
* Start beeline
* Run
{code:java}
set hive.execution.engine=tez;{code}
* Run
{code:java}
CREATE TABLE emp(id int, name string, department string, salary float) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS PARQUET TBLPROPERTIES ("parquet.compression"="zstd"); {code}
* Run
{code:java}
Insert into emp VALUES (1, 'some name', 'some dept', 1.15); {code}
* Getting
java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool
Java call stack for the error:
{code:java}
java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool
E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:90)
E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83)
E at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:111)
E at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236)
E at org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.<init>(BaseVectorizedColumnReader.java:137)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.<init>(VectorizedPrimitiveColumnReader.java:58)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:515)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:446)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:406)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:347)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:95)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:376)
E at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59)
E at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
E at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
E at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
E at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
E at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
E at java.security.AccessController.doPrivileged(Native Method)
E at javax.security.auth.Subject.doAs(Subject.java:422)
E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
E at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
E at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
E at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
E at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
E at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:748)
E Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.RecyclingBufferPool
E at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
E ... 36 more
E , errorMessage=Cannot recover from this error:java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool
E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:90)
E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83)
E at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:111)
E at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236)
E at org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.<init>(BaseVectorizedColumnReader.java:137)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.<init>(VectorizedPrimitiveColumnReader.java:58)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:515)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:446)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:406)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:347)
E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:95)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:376)
E at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119)
E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59)
E at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
E at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
E at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
E at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
E at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
E at java.security.AccessController.doPrivileged(Native Method)
E at javax.security.auth.Subject.doAs(Subject.java:422)
E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
E at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
E at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
E at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
E at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
E at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:748)
E Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.RecyclingBufferPool
E at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
E at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
E ... 36 more{code}
This is what happens during the execution of this insert statement:
# At some point Hive calls Tez execution on YARN cluster
# Tez calls the code in Hive-Exec jar. Tez classpath does not have zstd-jni.jar. Tez classpath include Tez jars, Tez lib jars and hive-exec.
# Hive code calls Parquet shaded in hive-exec.
# Parquet is failing because it cannot find zstd-jni classes at runtime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)