You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2012/07/18 19:47:34 UTC
[jira] [Commented] (KAFKA-406) Gzipped payload is a fully wrapped
Message (with headers), not just payload
[ https://issues.apache.org/jira/browse/KAFKA-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417304#comment-13417304 ]
Neha Narkhede commented on KAFKA-406:
-------------------------------------
Yes, this was intentional. The idea was to represent a compressed set of message just like a kafka Message. For this, we compress several messages into an array of bytes and treat it as the payload of a Message with compression codec GZIP or Snappy. This makes it easier to reason about compressed messages, since everything is a Message with a byte in its header telling us whether it is compressed or not.
> Gzipped payload is a fully wrapped Message (with headers), not just payload
> ---------------------------------------------------------------------------
>
> Key: KAFKA-406
> URL: https://issues.apache.org/jira/browse/KAFKA-406
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.1
> Environment: N/A
> Reporter: Lorenzo Alberton
>
> When creating a gzipped MessageSet, the collection of Messages is passed to CompressionUtils.compress(), where each message is serialised [1] into a buffer (not just the payload, the full Message with headers, uncompressed), then gripped, and finally wrapped into another Message [2].
> In other words, the consumer has to unwrap the Message flagged as gzipped, unzip the payload, and unwrap the unzipped payload again as a non-compressed Message.
> Is this double-wrapping the intended behaviour?
> [1] messages.foreach(m => m.serializeTo(messageByteBuffer))
> [2] new Message(outputStream.toByteArray, compressionCodec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira