You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "emkornfield (via GitHub)" <gi...@apache.org> on 2026/03/25 23:12:12 UTC
[PR] feat!: Enhance compression codec enum. [iceberg-rust]
emkornfield opened a new pull request, #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288
## Which issue does this PR close?
This is an intermediate PR for #1731
I'm splitting out changes from https://github.com/apache/iceberg-rust/pull/1851 to the compression codec to make it easier to review. Once we decide on approach here and merge it I'll update https://github.com/apache/iceberg-rust/pull/1851 accordingly.
## What changes are included in this PR?
- Add optional compression level to gzip and zstd (needed for when avro compression usage).
- Add Snappy as a compression codec (also will be used for Avro)
- Manually code up some previously auto-generated methods as a result.
AI helped with an initial version of this PR.
## Are these changes tested?
Additional unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003510433
##########
crates/iceberg/src/spec/table_properties.rs:
##########
@@ -85,13 +85,15 @@ pub(crate) fn parse_metadata_file_compression(
// Validate that only None and Gzip are used for metadata
match codec {
- CompressionCodec::None | CompressionCodec::Gzip => Ok(codec),
- CompressionCodec::Lz4 | CompressionCodec::Zstd => Err(Error::new(
- ErrorKind::DataInvalid,
- format!(
- "Invalid metadata compression codec: {value}. Only 'none' and 'gzip' are supported for metadata files."
- ),
- )),
+ CompressionCodec::None | CompressionCodec::Gzip(_) => Ok(codec),
+ CompressionCodec::Lz4 | CompressionCodec::Zstd(_) | CompressionCodec::Snappy => {
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#issuecomment-4136567155
@liurenjie1024 @blackmwk would appreciate your thoughts on this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3014829155
##########
crates/iceberg/src/puffin/writer.rs:
##########
@@ -251,7 +251,8 @@ mod tests {
async fn test_write_zstd_compressed_metric_data() {
let temp_dir = TempDir::new().unwrap();
let blobs = vec![blob_0(), blob_1()];
- let blobs_with_compression = blobs_with_compression(blobs.clone(), CompressionCodec::Zstd);
+ let blobs_with_compression =
+ blobs_with_compression(blobs.clone(), CompressionCodec::Zstd(None));
Review Comment:
I was thinking not changing testing behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003655446
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
Review Comment:
actually, thats not clear and this keeps existing behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003484196
##########
crates/iceberg/src/compression.rs:
##########
@@ -49,13 +94,17 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 decompression is not supported currently",
)),
- CompressionCodec::Zstd => Ok(zstd::stream::decode_all(&bytes[..])?),
- CompressionCodec::Gzip => {
+ CompressionCodec::Zstd(_) => Ok(zstd::stream::decode_all(&bytes[..])?),
+ CompressionCodec::Gzip(_) => {
let mut decoder = GzDecoder::new(&bytes[..]);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed)?;
Ok(decompressed)
}
+ CompressionCodec::Snappy => Err(Error::new(
+ ErrorKind::FeatureUnsupported,
+ "Snappy decompression is not supported currently",
+ )),
Review Comment:
This makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3012397248
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -3618,7 +3618,7 @@ mod tests {
let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json");
let json = serde_json::to_string(&original_metadata).unwrap();
- let compressed = CompressionCodec::Gzip
+ let compressed = CompressionCodec::Gzip(None)
Review Comment:
9 I think is max decompression level, 6 (is the default https://docs.rs/flate2/latest/src/flate2/lib.rs.html#251), by assigning None here I think we end up with 6. I'm in the process of changing the code to have a default constructor without the None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003403942
##########
crates/iceberg/src/compression.rs:
##########
@@ -49,13 +94,17 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 decompression is not supported currently",
)),
- CompressionCodec::Zstd => Ok(zstd::stream::decode_all(&bytes[..])?),
- CompressionCodec::Gzip => {
+ CompressionCodec::Zstd(_) => Ok(zstd::stream::decode_all(&bytes[..])?),
+ CompressionCodec::Gzip(_) => {
let mut decoder = GzDecoder::new(&bytes[..]);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed)?;
Ok(decompressed)
}
+ CompressionCodec::Snappy => Err(Error::new(
+ ErrorKind::FeatureUnsupported,
+ "Snappy decompression is not supported currently",
+ )),
Review Comment:
As far as I now the only place Snappy is used is within Avro, and Avro has its own code for doing the compression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003512357
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
Review Comment:
we can just use 0 here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3007318913
##########
crates/iceberg/src/compression.rs:
##########
@@ -17,28 +17,74 @@
//! Compression codec support for data compression and decompression.
+use std::fmt;
use std::io::{Read, Write};
use flate2::Compression;
use flate2::read::GzDecoder;
use flate2::write::GzEncoder;
-use serde::{Deserialize, Serialize};
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
use crate::{Error, ErrorKind, Result};
/// Data compression formats
-#[derive(Debug, PartialEq, Eq, Clone, Copy, Default, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
+#[derive(Debug, PartialEq, Eq, Clone, Copy, Default)]
pub enum CompressionCodec {
#[default]
/// No compression
None,
/// LZ4 single compression frame with content size present
Lz4,
- /// Zstandard single compression frame with content size present
- Zstd,
- /// Gzip compression
- Gzip,
+ /// Zstandard single compression frame with content size present. Optional level 0–22,
+ /// where 0 means default compression level (not no compression, unlike Gzip).
+ Zstd(Option<u8>),
Review Comment:
If there is no `no compression` option, why this has to be Option?
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
encoder.include_checksum(true)?;
encoder.set_pledged_src_size(Some(bytes.len().try_into()?))?;
std::io::copy(&mut &bytes[..], &mut encoder)?;
Ok(encoder.finish()?)
}
- CompressionCodec::Gzip => {
- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
+ CompressionCodec::Gzip(level) => {
+ let compression =
+ level.map_or_else(Compression::default, |l| Compression::new(l.min(9) as u32));
Review Comment:
Ditto.
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -3618,7 +3618,7 @@ mod tests {
let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json");
let json = serde_json::to_string(&original_metadata).unwrap();
- let compressed = CompressionCodec::Gzip
+ let compressed = CompressionCodec::Gzip(None)
Review Comment:
I think this is incorrect? It should be 9?
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
Review Comment:
Not related to this pr, but I don't think we should use a magic number here. We should create a constant.
##########
crates/iceberg/src/puffin/writer.rs:
##########
@@ -251,7 +251,8 @@ mod tests {
async fn test_write_zstd_compressed_metric_data() {
let temp_dir = TempDir::new().unwrap();
let blobs = vec![blob_0(), blob_1()];
- let blobs_with_compression = blobs_with_compression(blobs.clone(), CompressionCodec::Zstd);
+ let blobs_with_compression =
+ blobs_with_compression(blobs.clone(), CompressionCodec::Zstd(None));
Review Comment:
This is incorrect, similar to the one with gzip.
##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -3618,7 +3618,7 @@ mod tests {
let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json");
let json = serde_json::to_string(&original_metadata).unwrap();
- let compressed = CompressionCodec::Gzip
+ let compressed = CompressionCodec::Gzip(None)
Review Comment:
If 9 is default compression level, you could create a method `CompressionCodec::gzip_default` for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003339106
##########
crates/iceberg/src/compression.rs:
##########
@@ -17,28 +17,73 @@
//! Compression codec support for data compression and decompression.
+use std::fmt;
use std::io::{Read, Write};
use flate2::Compression;
use flate2::read::GzDecoder;
use flate2::write::GzEncoder;
-use serde::{Deserialize, Serialize};
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
use crate::{Error, ErrorKind, Result};
/// Data compression formats
-#[derive(Debug, PartialEq, Eq, Clone, Copy, Default, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
+#[derive(Debug, PartialEq, Eq, Clone, Copy, Default)]
pub enum CompressionCodec {
#[default]
/// No compression
None,
/// LZ4 single compression frame with content size present
Lz4,
- /// Zstandard single compression frame with content size present
- Zstd,
- /// Gzip compression
- Gzip,
+ /// Zstandard single compression frame with content size present. Optional level 0–22.
Review Comment:
nit: We should mention that level 0 for zstd means default compression level but not no compression like Gzip
##########
crates/iceberg/src/spec/table_properties.rs:
##########
@@ -85,13 +85,15 @@ pub(crate) fn parse_metadata_file_compression(
// Validate that only None and Gzip are used for metadata
match codec {
- CompressionCodec::None | CompressionCodec::Gzip => Ok(codec),
- CompressionCodec::Lz4 | CompressionCodec::Zstd => Err(Error::new(
- ErrorKind::DataInvalid,
- format!(
- "Invalid metadata compression codec: {value}. Only 'none' and 'gzip' are supported for metadata files."
- ),
- )),
+ CompressionCodec::None | CompressionCodec::Gzip(_) => Ok(codec),
+ CompressionCodec::Lz4 | CompressionCodec::Zstd(_) | CompressionCodec::Snappy => {
Review Comment:
nit: since we only support `None` and `Gzip`, this can be `_ =>` to fail all other cases
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +115,24 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
encoder.include_checksum(true)?;
encoder.set_pledged_src_size(Some(bytes.len().try_into()?))?;
std::io::copy(&mut &bytes[..], &mut encoder)?;
Ok(encoder.finish()?)
}
- CompressionCodec::Gzip => {
- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
+ CompressionCodec::Gzip(level) => {
+ let compression = Compression::new(level.unwrap_or(6).min(9) as u32);
Review Comment:
We can just call `Compression::default()` when `level` is `None`
##########
crates/iceberg/src/compression.rs:
##########
@@ -49,13 +94,17 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 decompression is not supported currently",
)),
- CompressionCodec::Zstd => Ok(zstd::stream::decode_all(&bytes[..])?),
- CompressionCodec::Gzip => {
+ CompressionCodec::Zstd(_) => Ok(zstd::stream::decode_all(&bytes[..])?),
+ CompressionCodec::Gzip(_) => {
let mut decoder = GzDecoder::new(&bytes[..]);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed)?;
Ok(decompressed)
}
+ CompressionCodec::Snappy => Err(Error::new(
+ ErrorKind::FeatureUnsupported,
+ "Snappy decompression is not supported currently",
+ )),
Review Comment:
Do we plan to address this in a follow up PR? If so, could you create a tracking issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3012497125
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
Review Comment:
moved to a private constant used in default construction.
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +116,25 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
encoder.include_checksum(true)?;
encoder.set_pledged_src_size(Some(bytes.len().try_into()?))?;
std::io::copy(&mut &bytes[..], &mut encoder)?;
Ok(encoder.finish()?)
}
- CompressionCodec::Gzip => {
- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
+ CompressionCodec::Gzip(level) => {
+ let compression =
+ level.map_or_else(Compression::default, |l| Compression::new(l.min(9) as u32));
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003509871
##########
crates/iceberg/src/compression.rs:
##########
@@ -66,19 +115,24 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 compression is not supported currently",
)),
- CompressionCodec::Zstd => {
+ CompressionCodec::Zstd(level) => {
let writer = Vec::<u8>::new();
- let mut encoder = zstd::stream::Encoder::new(writer, 3)?;
+ let mut encoder = zstd::stream::Encoder::new(writer, level.unwrap_or(3) as i32)?;
encoder.include_checksum(true)?;
encoder.set_pledged_src_size(Some(bytes.len().try_into()?))?;
std::io::copy(&mut &bytes[..], &mut encoder)?;
Ok(encoder.finish()?)
}
- CompressionCodec::Gzip => {
- let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
+ CompressionCodec::Gzip(level) => {
+ let compression = Compression::new(level.unwrap_or(6).min(9) as u32);
Review Comment:
done.
##########
crates/iceberg/src/compression.rs:
##########
@@ -17,28 +17,73 @@
//! Compression codec support for data compression and decompression.
+use std::fmt;
use std::io::{Read, Write};
use flate2::Compression;
use flate2::read::GzDecoder;
use flate2::write::GzEncoder;
-use serde::{Deserialize, Serialize};
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
use crate::{Error, ErrorKind, Result};
/// Data compression formats
-#[derive(Debug, PartialEq, Eq, Clone, Copy, Default, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
+#[derive(Debug, PartialEq, Eq, Clone, Copy, Default)]
pub enum CompressionCodec {
#[default]
/// No compression
None,
/// LZ4 single compression frame with content size present
Lz4,
- /// Zstandard single compression frame with content size present
- Zstd,
- /// Gzip compression
- Gzip,
+ /// Zstandard single compression frame with content size present. Optional level 0–22.
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3003403942
##########
crates/iceberg/src/compression.rs:
##########
@@ -49,13 +94,17 @@ impl CompressionCodec {
ErrorKind::FeatureUnsupported,
"LZ4 decompression is not supported currently",
)),
- CompressionCodec::Zstd => Ok(zstd::stream::decode_all(&bytes[..])?),
- CompressionCodec::Gzip => {
+ CompressionCodec::Zstd(_) => Ok(zstd::stream::decode_all(&bytes[..])?),
+ CompressionCodec::Gzip(_) => {
let mut decoder = GzDecoder::new(&bytes[..]);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed)?;
Ok(decompressed)
}
+ CompressionCodec::Snappy => Err(Error::new(
+ ErrorKind::FeatureUnsupported,
+ "Snappy decompression is not supported currently",
+ )),
Review Comment:
As far as I know the only place Snappy is used is within Avro, and Avro has its own code for doing the compression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!: Enhance compression codec enum. [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #2288:
URL: https://github.com/apache/iceberg-rust/pull/2288#discussion_r3012424232
##########
crates/iceberg/src/puffin/writer.rs:
##########
@@ -251,7 +251,8 @@ mod tests {
async fn test_write_zstd_compressed_metric_data() {
let temp_dir = TempDir::new().unwrap();
let blobs = vec![blob_0(), blob_1()];
- let blobs_with_compression = blobs_with_compression(blobs.clone(), CompressionCodec::Zstd);
+ let blobs_with_compression =
+ blobs_with_compression(blobs.clone(), CompressionCodec::Zstd(None));
Review Comment:
I'm sorry could you elaborate on why you think was incorrect, it appears to test only code and the actual compress call would use the default value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org