You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 李斌松 <li...@gmail.com> on 2018/12/20 03:38:00 UTC
[Spark SQL]use zstd, No enum constant parquet.hadoop.metadata.CompressionCodecName.ZSTD
Import parquet-hadoop-bundle jar. into the spark hive project When you
compress data using zstd, you may load it preferentially from the
parquet-hadoop-bundle, and you canundefinedt find the enum constant
parquet.hadoop.metadata.CompressionCodecName.ZSTD
>
> 18/12/20 10:35:28 ERROR Executor: Exception in task 0.2 in stage 1.0 (TID
> 5)
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: No enum constant
> parquet.hadoop.metadata.CompressionCodecName.ZSTD
> /parquet
> at
> org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:109)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:243)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:175)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:174)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:406)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:412)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: No enum constant
> parquet.hadoop.metadata.CompressionCodecName.ZSTD
> at java.lang.Enum.valueOf(Enum.java:238)
> at
> parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:24)
> at
> parquet.hadoop.metadata.CompressionCodecName.fromConf(CompressionCodecName.java:34)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.initializeSerProperties(ParquetRecordWriterWrapper.java:94)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:61)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:125)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:114)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:261)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:246)
> ... 15 more
Re:Re: [Spark SQL]use zstd, No enum constant
parquet.hadoop.metadata.CompressionCodecName.ZSTD
Posted by 大啊 <be...@163.com>.
I think your hive table using CompressionCodecName, but your
parquet-hadoop-bundle.jar in spark classpaths is not a correct version.
At 2018-12-21 12:07:22, "Jiaan Geng" <be...@163.com> wrote:
>I think your hive table using CompressionCodecName, but your
>parquet-hadoop-bundle.jar in spark classpaths is not a correct version.
>
>
>
>--
>Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
>---------------------------------------------------------------------
>To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [Spark SQL]use zstd, No enum constant
parquet.hadoop.metadata.CompressionCodecName.ZSTD
Posted by Jiaan Geng <be...@163.com>.
I think your hive table using CompressionCodecName, but your
parquet-hadoop-bundle.jar in spark classpaths is not a correct version.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org