[jira] [Commented] (SPARK-19112) add codec for ZStandard

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811935#comment-15811935 ] 

Thomas Graves commented on SPARK-19112:
---------------------------------------

For any sort of Hadoop output formats we get it for free but for the internal and shuffle outputs I think we have to add support to spark.io.compression.codec for this.    

I'm also wondering if there are other output formats like parquet that we would want to add this option to (spark.sql.parquet.compression.codec), but I haven't investigated those enough.   

> add codec for ZStandard
> -----------------------
>
>                 Key: SPARK-19112
>                 URL: https://issues.apache.org/jira/browse/SPARK-19112
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Thomas Graves
>            Priority: Minor
>
> ZStandard: https://github.com/facebook/zstd and http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was recently released. Hadoop (https://issues.apache.org/jira/browse/HADOOP-13578) and others (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it.
> Zstd seems to give great results => Gzip level Compression with Lz4 level CPU.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org