You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mattefunnel (via GitHub)" <gi...@apache.org> on 2026/04/27 20:09:03 UTC
[PR] Honor parquet compression table properties [iceberg-rust]
mattefunnel opened a new pull request, #2378:
URL: https://github.com/apache/iceberg-rust/pull/2378
Adds typed Parquet compression table properties and uses them for DataFusion writes.
Includes data/delete codec and level resolution, ZSTD fallback for absent or unrecognized values, and DataFusion write tests that inspect Parquet metadata.
Checks: cargo fmt --all; cargo test -p iceberg spec::table_properties::tests; cargo test -p iceberg-datafusion test_iceberg_write_exec_ -- --nocapture; cargo check -p iceberg-datafusion. make test could not run because docker is not installed in this environment.
Co-Authored-By: Claude <no...@anthropic.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Honor parquet compression table properties [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2378:
URL: https://github.com/apache/iceberg-rust/pull/2378#issuecomment-4559851010
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Honor parquet compression table properties [iceberg-rust]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #2378:
URL: https://github.com/apache/iceberg-rust/pull/2378#discussion_r3149984694
##########
crates/iceberg/src/spec/table_properties.rs:
##########
@@ -204,35 +144,108 @@ impl TableProperties {
pub const PROPERTY_DELETE_DEFAULT_FILE_FORMAT: &str = "write.delete.format.default";
/// Default value for data file format
pub const PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT: &str = "parquet";
+ /// Default compression codec for Parquet data files.
+ pub const PROPERTY_WRITE_PARQUET_COMPRESSION_CODEC: &str = "write.parquet.compression-codec";
+ /// Default compression codec for Parquet delete files.
+ pub const PROPERTY_WRITE_DELETE_PARQUET_COMPRESSION_CODEC: &str =
+ "write.delete.parquet.compression-codec";
+ /// Default value for Parquet data file compression.
+ pub const PROPERTY_WRITE_PARQUET_COMPRESSION_CODEC_DEFAULT: &str = "zstd";
+ /// Default compression level for Parquet data files.
+ pub const PROPERTY_WRITE_PARQUET_COMPRESSION_LEVEL: &str = "write.parquet.compression-level";
+ /// Default compression level for Parquet delete files.
+ pub const PROPERTY_WRITE_DELETE_PARQUET_COMPRESSION_LEVEL: &str =
+ "write.delete.parquet.compression-level";
/// Target file size for newly written files.
pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES: &str = "write.target-file-size-bytes";
/// Default target file size
pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT: usize = 512 * 1024 * 1024; // 512 MB
-
- /// Compression codec for metadata files (JSON)
- pub const PROPERTY_METADATA_COMPRESSION_CODEC: &str = "write.metadata.compression-codec";
- /// Default metadata compression codec - uncompressed
- pub const PROPERTY_METADATA_COMPRESSION_CODEC_DEFAULT: &str = "none";
/// Whether to use `FanoutWriter` for partitioned tables (handles unsorted data).
/// If false, uses `ClusteredWriter` (requires sorted data, more memory efficient).
pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled";
/// Default value for fanout writer enabled
pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT: bool = true;
- /// Property key for enabling garbage collection on drop.
- /// When set to `false`, data files will not be deleted when a table is dropped.
- /// Defaults to `true`.
- pub const PROPERTY_GC_ENABLED: &str = "gc.enabled";
- /// Default value for gc.enabled
- pub const PROPERTY_GC_ENABLED_DEFAULT: bool = true;
+ /// Returns the Parquet compression codec requested by table properties.
+ ///
+ /// Unknown or unparsable values intentionally fall back to ZSTD so table
+ /// writes remain possible even when catalogs provide unexpected strings.
+ pub fn write_parquet_compression(&self) -> Compression {
+ parse_parquet_compression(
+ &self.write_parquet_compression_codec,
+ self.write_parquet_compression_level.as_deref(),
+ )
+ .unwrap_or(Self::default_parquet_compression())
+ }
+
+ /// Returns the Parquet compression codec requested for delete files.
+ ///
+ /// Mirrors Java's write-conf behavior: delete compression inherits data
+ /// compression unless `write.delete.parquet.compression-codec` is set.
+ pub fn write_delete_parquet_compression(&self) -> Compression {
+ parse_parquet_compression(
+ &self.write_delete_parquet_compression_codec,
+ self.write_delete_parquet_compression_level.as_deref(),
+ )
+ .unwrap_or(Self::default_parquet_compression())
Review Comment:
Shouldn't this fallback to `self.write_parquet_compression()` as https://github.com/apache/iceberg-rust/pull/2378/changes#diff-b3937e8d5290e685464b83957bdce1758ebe4ce122e25b12edcffd76280582afR247 does ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Honor parquet compression table properties [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2378:
URL: https://github.com/apache/iceberg-rust/pull/2378#issuecomment-4617991237
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Honor parquet compression table properties [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2378: Honor parquet compression table properties
URL: https://github.com/apache/iceberg-rust/pull/2378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org