Re: [I] Respect Spark compression settings in native Parquet writer [datafusion-comet]

["dwsmith1983 (via GitHub)" <gi...@apache.org>]

dwsmith1983 commented on issue #2814:
URL: https://github.com/apache/datafusion-comet/issues/2814#issuecomment-5637961390

   The native Parquet writer now honors Spark's codec configuration. `CometDataWritingCommand` (spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala) resolves the codec with Spark's own precedence, the `compression` write option, then `parquet.compression`, then `spark.sql.parquet.compression.codec`, and accepts none/uncompressed, snappy, lz4, zstd and gzip (gzip landed last, in #4930, c65a5ee80); native/core/src/execution/operators/parquet_writer.rs maps each to the parquet-rs codec and carries the compression level for zstd. An lz4 level is not something parquet-rs exposes (`Compression::LZ4` takes no level), so that part of the ask has nothing to configure. Suggest closing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org