You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/09/16 03:53:00 UTC
[jira] [Assigned] (SPARK-36773) Fix the uts to check the parquet
compression
[ https://issues.apache.org/jira/browse/SPARK-36773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-36773:
------------------------------------
Assignee: (was: Apache Spark)
> Fix the uts to check the parquet compression
> --------------------------------------------
>
> Key: SPARK-36773
> URL: https://issues.apache.org/jira/browse/SPARK-36773
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3
> Reporter: jingpan xiong
> Priority: Major
> Fix For: 3.3
>
>
> In my own test env. I import a wrong jar version about zstd-jni-1.4.4-3.jar with the parquet version 1.12.0, when I pass the full uts and release a spark version to my env , using the parquet table with compression zstd, it throw a "NoClassDefFoundError" Exception. And I check the parquet source code, find the right zstd version for the 1.12.0 is 1.4.9-1. Once I upgrade the version about zstd-jni, it work well.
> !https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2021/8/14/xwx915262/ed1735e06a684bf19cf1d1a68a130b6a/image.png!
> But I think we need check for all plugin compression algorithm can work well when we upgrade the version about those compression jars and parquet version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org