[I] feat(connectors): add compression support to S3 sink connector (gzip, zstd) [iggy]

["atharvalade (via GitHub)" <gi...@apache.org>]

atharvalade opened a new issue, #3104:
URL: https://github.com/apache/iggy/issues/3104

   ## Description
   
   The S3 sink connector (`iggy_connector_s3_sink`) added in #2976 currently writes uncompressed files to S3. For production workloads with high message throughput, compression is essential to reduce storage costs and upload times.
   
   ## Proposed Changes
   
   Add a `compression` config option to the S3 sink connector supporting:
   
   - **`none`** (default, current behavior)
   - **`gzip`** — widely supported, good compatibility with downstream tools (Athena, Spark, etc.)
   - **`zstd`** — better compression ratio and speed, growing ecosystem support
   
   ### Config Example
   
   ```toml
   [plugin_config]
   compression = "gzip"  # none | gzip | zstd
   ```
   
   #### Implementation Notes
   
   - Compress the finalized buffer bytes before uploading to S3 (after finalize_buffer, before upload_with_retry)
   - Append the appropriate file extension (.jsonl.gz, .jsonl.zst, .json.gz, etc.) — the path module already derives extensions from OutputFormat, this needs to account for compression
   - Set the correct Content-Encoding header on the S3 put_object call
   - Use flate2 for gzip and zstd crate for zstd — check if they're already in the workspace, otherwise add them
   - Add unit tests for round-trip compress/decompress verification
   - Update the connector README and config.toml example


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] feat(connectors): add compression support to S3 sink connector (gzip, zstd) [iggy]

["drifting-shard (via GitHub)" <gi...@apache.org>]

drifting-shard commented on issue #3104:
URL: https://github.com/apache/iggy/issues/3104#issuecomment-5396683731

   Hi @atharvalade, are you still working on this? If not, I’d be interested in taking it. I’ll first validate the current finalize-buffer → S3 upload path, then propose a minimal gzip/zstd implementation with round-trip coverage and unchanged none behavior. Maintainers, may I proceed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] feat(connectors): add compression support to S3 sink connector (gzip, zstd) [iggy]

["justinmclean (via GitHub)" <gi...@apache.org>]

justinmclean commented on issue #3104:
URL: https://github.com/apache/iggy/issues/3104#issuecomment-6329569247

   @atharvalade are you still planning to do this? @drifting-shard has offered to take it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] feat(connectors): add compression support to S3 sink connector (gzip, zstd) [iggy]

["atharvalade (via GitHub)" <gi...@apache.org>]

atharvalade commented on issue #3104:
URL: https://github.com/apache/iggy/issues/3104#issuecomment-4234480418

   Please assign this to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org