[jira] [Commented] (KAFKA-79) Introduce the compression feature in Kafka

["Jay Kreps (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119022#comment-13119022 ] 

Jay Kreps commented on KAFKA-79:
--------------------------------

Scott, we would gladly take that feature! The compression is pluggable so adding that should not be too hard, we just need another compression id for snappy (gzip=1, so snappy=2). We should still keep gzip, I think it has a place, for example in our usage our biggest bottleneck is inter-datacenter bandwidth, so we will probably stay with gzip I think, plus it has no native depenencies so it is a little better "out-of-the-box" experience.

One thing we should think through is how this is packaged. Typically we make these things contrib/ packages to avoid adding to the main dependencies. It doesn't appear that the snappy jar has any external dependencies though and it looks like it packages up the .so/.dll files for all platforms in its jar, which is nice. I think it might be better just to add it in the main code, and ensure that none of the snappy classes are loaded unless the user selects snappy as the compression type (this will make the snappy jar optional for people--you only need it if you use it).

Any other thoughts from people?

                
> Introduce the compression feature in Kafka
> ------------------------------------------
>
>                 Key: KAFKA-79
>                 URL: https://issues.apache.org/jira/browse/KAFKA-79
>             Project: Kafka
>          Issue Type: New Feature
>    Affects Versions: 0.6
>            Reporter: Neha Narkhede
>             Fix For: 0.7
>
>
> With this feature, we can enable end-to-end block compression in Kafka. The idea is to enable compression on the producer for some or all topics, write the data in compressed format on the server and make the consumers compression aware. The data will be decompressed only on the consumer side. Ideally, there should be a choice of compression codecs to be used by the producer. That means a change to the message header as well as the network byte format. On the consumer side, the state maintenance behavior of the zookeeper consumer changes. For compressed data, the consumed offset will be advanced one compressed message at a time. For uncompressed data, consumed offset will be advanced one message at a time. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira