You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2018/01/06 10:23:03 UTC
[jira] [Resolved] (SPARK-21786) The
'spark.sql.parquet.compression.codec' configuration doesn't take effect on
tables with partition field(s)
[ https://issues.apache.org/jira/browse/SPARK-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-21786.
-----------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
> The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-21786
> URL: https://issues.apache.org/jira/browse/SPARK-21786
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Jinhua Fu
> Assignee: Jinhua Fu
> Fix For: 2.3.0
>
>
> For tables created like below, 'spark.sql.parquet.compression.codec' doesn't take any effect when insert data. And because the default compression codec is 'uncompressed', if I want to change the compression codec, I have to change it by 'set parquet.compression=xxxx'.
> Contrast，tables without any partition field will work normal with 'spark.sql.parquet.compression.codec',and the default compression codec is 'snappy', but it seems 'parquet.compression' no longer in effect.
> Should we use the ‘spark.sql.parquet.compression.codec’ configuration uniformly？
> ----------------------------------------------------------------------------------------------------
> CREATE TABLE Test_Parquet(provincecode int, citycode int, districtcode int)
> PARTITIONED BY (p_provincecode int)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE Test_Parquet select * from TableB;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org