You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/09/11 08:53:03 UTC
[jira] [Commented] (SPARK-21786) The
'spark.sql.parquet.compression.codec' configuration doesn't take effect on
tables with partition field(s)
[ https://issues.apache.org/jira/browse/SPARK-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160937#comment-16160937 ]
Apache Spark commented on SPARK-21786:
--------------------------------------
User 'fjh100456' has created a pull request for this issue:
https://github.com/apache/spark/pull/19189
> The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-21786
> URL: https://issues.apache.org/jira/browse/SPARK-21786
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Jinhua Fu
>
> For tables created like below, 'spark.sql.parquet.compression.codec' doesn't take any effect when insert data. And because the default compression codec is 'uncompressed', if I want to change the compression codec, I have to change it by 'set parquet.compression=xxxx'.
> Contrast，tables without any partition field will work normal with 'spark.sql.parquet.compression.codec',and the default compression codec is 'snappy', but it seems 'parquet.compression' no longer in effect.
> Should we use the ‘spark.sql.parquet.compression.codec’ configuration uniformly？
> ----------------------------------------------------------------------------------------------------
> CREATE TABLE Test_Parquet(provincecode int, citycode int, districtcode int)
> PARTITIONED BY (p_provincecode int)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE Test_Parquet select * from TableB;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org