[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression

["Evan Huus (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809464#comment-13809464 ] 

Evan Huus commented on KAFKA-1110:
----------------------------------

I am the primary author of the golang library and the one quoted in the main description. The "[1]" link in my quote was a reference to the Snappy framing format spec: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

The part that is confusing me the most is the 0x82 at the beginning of the message produced by the ConsoleConsumer, since it does not match any possible structure I know of. Raw snappy would expect a little-endian varint of the message length (which 0x82 is not), snappy framing would expect 0xff prior to sNaPpY (as opposed to the actual SNAPPY in the message) and a Kafka string would be a two-byte length.

Worth noting: recent development builds of Wireshark support decoding the Kafka protocol as described in the spec, so that may aid in reading the PCAP files for debugging. You will have to set a preference to tell it which port to decode on though.

> Unable to produce messages with snappy/gzip compression
> -------------------------------------------------------
>
>                 Key: KAFKA-1110
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1110
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>         Environment: Kafka version: kafka-0.8.0-beta1
> OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64
>            Reporter: Arup Malakar
>         Attachments: kafka_producer_snappy_pkt_63.pcapng, sarama_producer_snappy_pkt_1.pcapng
>
>
> Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is following the specs as defined in: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol but messages are not getting into the kafka log file and consumers never see them when gzip/snappy is used. Without compression it works fine though.
> Few observations we made:
> 1. Kafka service does have required jars to be able to interpret snappy messages. When I modify ConsoleProducer to produce messages using   SnappyCompressionCodec instead of default GZip codec. I was able to produce/consume messages. Looking at the kafka log files I see that Snappy Compression was indeed getting used:
> % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/aruptest-0/00000000000000000000.log | tail -1
> offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 compresscodec: SnappyCompressionCodec crc: 1602790249
> So I don't think it would be a case of missing jars in kafka server.
> 2. Kafka doesn't return any error if the message doesn't make it to the log file. This seems pretty serious, as I would expect kafka to throw an error if I am using WaitForLocal/WaitForAll.
> 3. We did an inspection of the tcp packet to see the difference between what ConsoleProducer sends vs what sarama sends
> (Following is a copy/paste from a github issue):
> [~eapache] : 
> So I have no idea what the ConsoleProducer is actually sending in this case. The outer protocol layers in both cases look identical, but if you compare the actual message value:
> a. Sarama sends two bytes of snappy header and then "<msg-payload>" (since Snappy decides it's too short to properly encode, so makes it a literal). Pretty straightforward.
> b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what appears to be a complete embedded produce request without any compression. This is neither valid snappy nor valid Kafka according to anything I've seen, so I'm pretty confused. It looks almost like an incorrect version of [1] but it's missing several key fields and the case of the identifying string is wrong.
> Let us know if recent changes in the codebase makes the protocol page obsolete, in that case if the protocol page is updated we could update our client to use the new spec.
> More information could be found in the following github issue: https://github.com/Shopify/sarama/issues/32



--
This message was sent by Atlassian JIRA
(v6.1#6144)