You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Denys Kuzmenko (Jira)" <ji...@apache.org> on 2026/10/02 12:23:00 UTC
[jira] [Updated] (HIVE-30129) Iceberg: HiveIcebergOutputFormat silently drops the cluster's parquet.compression override, writing Iceberg's own default codec instead
[ https://issues.apache.org/jira/browse/HIVE-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denys Kuzmenko updated HIVE-30129:
----------------------------------
Description:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's parquet.compression into Iceberg's write.parquet.compression-codec for any table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION, jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even when it's set cluster-wide in hive-site.xml — it isn't a registered HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's config is. The bridge silently never fires, and the write falls through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error and no trace in SHOW TBLPROPERTIES.
*Impact*: any Iceberg table that doesn't set its own write.parquet.compression-codec TBLPROPERTIES silently ignores the cluster-wide codec setting and gets Iceberg's default instead — defeating the only mechanism Hive has for pinning one codec across Parquet and Iceberg tables.
was:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's parquet.compression into Iceberg's write.parquet.compression-codec for any table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION, jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even when it's set cluster-wide in hive-site.xml — it isn't a registered HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's config is. The bridge silently never fires, and the write falls through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error and no trace in SHOW TBLPROPERTIES.
> Iceberg: HiveIcebergOutputFormat silently drops the cluster's parquet.compression override, writing Iceberg's own default codec instead
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30129
> URL: https://issues.apache.org/jira/browse/HIVE-30129
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's parquet.compression into Iceberg's write.parquet.compression-codec for any table that sets no codec of its own:
>
> {code}
> if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
> jc.get(ParquetOutputFormat.COMPRESSION) != null) {
> table.properties().put(TableProperties.PARQUET_COMPRESSION, jc.get(ParquetOutputFormat.COMPRESSION));
> }
> {code}
>
> In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even when it's set cluster-wide in hive-site.xml — it isn't a registered HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's config is. The bridge silently never fires, and the write falls through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error and no trace in SHOW TBLPROPERTIES.
> *Impact*: any Iceberg table that doesn't set its own write.parquet.compression-codec TBLPROPERTIES silently ignores the cluster-wide codec setting and gets Iceberg's default instead — defeating the only mechanism Hive has for pinning one codec across Parquet and Iceberg tables.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)