You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "770120041 (via GitHub)" <gi...@apache.org> on 2026/03/09 00:34:26 UTC
[PR] Core: Update default Parquet compression codec to ZSTD [iceberg]
770120041 opened a new pull request, #15549:
URL: https://github.com/apache/iceberg/pull/15549
## Summary
Update the default Parquet compression codec from GZIP to ZSTD to align with the Iceberg spec v1.4.0 recommendation.
ZSTD provides better compression ratios and faster decompression compared to GZIP, and has been the recommended default since spec version 1.4.0.
Fixes #15236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Update default Parquet compression codec to ZSTD [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #15549:
URL: https://github.com/apache/iceberg/pull/15549#discussion_r2904700765
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -143,7 +143,7 @@ private TableProperties() {}
public static final String PARQUET_COMPRESSION = "write.parquet.compression-codec";
public static final String DELETE_PARQUET_COMPRESSION = "write.delete.parquet.compression-codec";
- public static final String PARQUET_COMPRESSION_DEFAULT = "gzip";
+ public static final String PARQUET_COMPRESSION_DEFAULT = "zstd";
public static final String PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0 = "zstd";
Review Comment:
> Update the default Parquet compression codec from GZIP to ZSTD to align with the Iceberg spec v1.4.0 recommendation.
The default compression since v1.4.0 is already zstd, as PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0 shows, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Update default Parquet compression codec to ZSTD [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #15549:
URL: https://github.com/apache/iceberg/pull/15549#issuecomment-4028825100
This PR appears to be generated and submitted by AI without human review. Closing for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Update default Parquet compression codec to ZSTD [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #15549: Core: Update default Parquet compression codec to ZSTD
URL: https://github.com/apache/iceberg/pull/15549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Update default Parquet compression codec to ZSTD [iceberg]
Posted by "770120041 (via GitHub)" <gi...@apache.org>.
770120041 commented on code in PR #15549:
URL: https://github.com/apache/iceberg/pull/15549#discussion_r2905808502
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -143,7 +143,7 @@ private TableProperties() {}
public static final String PARQUET_COMPRESSION = "write.parquet.compression-codec";
public static final String DELETE_PARQUET_COMPRESSION = "write.delete.parquet.compression-codec";
- public static final String PARQUET_COMPRESSION_DEFAULT = "gzip";
+ public static final String PARQUET_COMPRESSION_DEFAULT = "zstd";
public static final String PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0 = "zstd";
Review Comment:
Thanks for the review! You're right that new tables already get ZSTD via `persistedProperties()` in `TableMetadata`, which explicitly sets `write.parquet.compression-codec = zstd` at creation time.
However, `PARQUET_COMPRESSION_DEFAULT` is still used as the fallback in `Parquet.java`, `SparkWriteConf`, and `FlinkWriteConf` for tables that don't have the property set (e.g., tables created by other implementations like pyiceberg, or older tables). I see from the [issue discussion](https://github.com/apache/iceberg/issues/15236#issuecomment-3854510554) that this was intentional to avoid changing behavior for existing tables on upgrade.
Given that context, this change would affect existing tables that lack an explicit compression property, do we want to update those or we want to keep the existing behavior. I am open to close this pr if this is intended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org