[I] bug: gzip table metadata is neither written nor detected as gzip [iceberg-cpp]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #1013:
URL: https://github.com/apache/iceberg-cpp/issues/1013

   ## Summary
   
   With `write.metadata.compression-codec=gzip`, iceberg-cpp names new metadata files `<version>-<uuid>.gz.metadata.json`, the same layout Iceberg Java uses, but the round trip is broken on both sides:
   
   - `TableMetadataUtil::Write` never compresses. It writes plain JSON into the `.gz.metadata.json` file, so Java (or any reader that trusts the name) fails to inflate it.
   - `Codec::FromFileName` misdetects the codec. It calls `find_last_of(".metadata.json")`, which searches for any single character of that set, not the suffix. For `00001-abc.gz.metadata.json` it lands on the final `n`, the `.gz` check fails, and the file is read as uncompressed. A gzip file written by Java under that name therefore fails to parse.
   
   The two bugs mask each other inside iceberg-cpp: a file it writes is plain JSON and is read back as plain JSON, so its own round trip happens to work.
   
   ## Root cause
   
   `src/iceberg/table_metadata.cc`:
   
   ```cpp
   auto pos = file_name.find_last_of(kTableMetadataFileSuffix);  // character-set search
   ```
   
   and `TableMetadataUtil::Write(io, location, metadata)` serializes and calls `io.WriteFile` with no codec step. Only a `GZipDecompressor` exists in `util/gzip_internal.h`.
   
   Java's `TableMetadataParser.Codec.fromFileName` uses `lastIndexOf(".metadata.json")`, and `internalWrite` wraps the stream in `GZIPOutputStream` when that codec is gzip.
   
   ## Impact
   
   A table configured with gzip metadata compression is not readable across engines in either direction: Java cannot read what iceberg-cpp writes, and iceberg-cpp cannot read what Java writes. The default codec is `none`, so tables without that property are unaffected.
   
   The `find_last_of` bug also means `FromFileName` accepts names with no `.metadata.json` at all (e.g. `foo.txt` is classified as uncompressed), while Java rejects them.
   
   ## Proposed fix
   
   Use `rfind` in `FromFileName`, and compress in `Write` when the location's codec is gzip, with a small `GZipCompressor` next to the existing `GZipDecompressor`. Both halves have to land together: fixing only the read side makes iceberg-cpp try to inflate the plain JSON it writes today.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] bug: gzip table metadata is neither written nor detected as gzip [iceberg-cpp]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac closed issue #1013: bug: gzip table metadata is neither written nor detected as gzip
URL: https://github.com/apache/iceberg-cpp/issues/1013


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org