You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tanmayrauth (via GitHub)" <gi...@apache.org> on 2026/05/05 22:16:41 UTC
[PR] feat(table): support zstd metadata compression codec [iceberg-go]
tanmayrauth opened a new pull request, #1020:
URL: https://github.com/apache/iceberg-go/pull/1020
Adds MetadataCompressionCodecZstd alongside gzip/none and wires it through both read (decompress .metadata.json.zstd) and write (write.metadata.compression-codec=zstd) paths. Uses klauspost/compress/zstd already present as a transitive dependency.
Closes #1005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on code in PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#discussion_r3203457177
##########
catalog/internal/utils.go:
##########
@@ -55,10 +56,19 @@ func WriteTableMetadata(metadata table.Metadata, fs icebergio.WriteFileIO, loc s
var writer io.Writer = out
var compressWriter io.WriteCloser
- if compression == table.MetadataCompressionCodecGzip {
+ switch compression {
+ case table.MetadataCompressionCodecGzip:
compressWriter = gzip.NewWriter(out)
writer = compressWriter
defer internal.CheckedClose(compressWriter, &err)
+ case table.MetadataCompressionCodecZstd:
+ enc, zErr := zstd.NewWriter(out)
+ if zErr != nil {
+ return zErr
+ }
+ compressWriter = enc
+ writer = compressWriter
+ defer internal.CheckedClose(compressWriter, &err)
Review Comment:
Good catch, removed the intermediate variable. Also cleaned up the gzip case the same way while I was at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#issuecomment-4390111565
I'm trying to add comment reviews but GitHub is failng at this...
Essentially, we should create a more generic helper for this since the only difference between the gzip and zstd routes are essentially `gzip.NewReader` vs `zstd.NewReader`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#issuecomment-4400151055
@zeroshade the PR has been updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#discussion_r3203000997
##########
catalog/internal/utils.go:
##########
@@ -55,10 +56,19 @@ func WriteTableMetadata(metadata table.Metadata, fs icebergio.WriteFileIO, loc s
var writer io.Writer = out
var compressWriter io.WriteCloser
- if compression == table.MetadataCompressionCodecGzip {
+ switch compression {
+ case table.MetadataCompressionCodecGzip:
compressWriter = gzip.NewWriter(out)
writer = compressWriter
defer internal.CheckedClose(compressWriter, &err)
+ case table.MetadataCompressionCodecZstd:
+ enc, zErr := zstd.NewWriter(out)
+ if zErr != nil {
+ return zErr
+ }
+ compressWriter = enc
+ writer = compressWriter
+ defer internal.CheckedClose(compressWriter, &err)
Review Comment:
we don't need the `compressWriter` intermediate here.
we can just use `defer internal.CheckedClose(enc, &err)` right? `compressWriter` doesn't need to be pre-declared
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#issuecomment-4393005677
The spec's write.metadata.compression-codec formally lists only gzip and none, but both PyIceberg and the Java implementation already support zstd as an extension I've followed the same pattern. For the generic helper point, I've refactored it: extracted a newDecompressor(r io.Reader, codec string) (io.ReadCloser, error) that handles both codecs behind a unified io.ReadCloser interface, so adding a future codec is just one case in a switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(table): support zstd metadata compression codec [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #1020:
URL: https://github.com/apache/iceberg-go/pull/1020#issuecomment-4390087967
Does the Iceberg spec explicitly state what compression codecs are allow for metadata.json that has zstd part of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org