[jira] [Created] (KAFKA-4169) Calculation of message size is too conservative for compressed messages

["Dustin Cote (JIRA)" <ji...@apache.org>]

Dustin Cote created KAFKA-4169:
----------------------------------

             Summary: Calculation of message size is too conservative for compressed messages
                 Key: KAFKA-4169
                 URL: https://issues.apache.org/jira/browse/KAFKA-4169
             Project: Kafka
          Issue Type: Bug
          Components: producer 
    Affects Versions: 0.10.0.1
            Reporter: Dustin Cote


Currently the producer uses the uncompressed message size to check against {{max.request.size}} even if a {{compression.type}} is defined.  This can be reproduced as follows:

{code}
# dd if=/dev/zero of=/tmp/outsmaller.dat bs=1024 count=1000

# cat /tmp/out.dat | bin/kafka-console-producer --broker-list localhost:9092 --topic tester --producer-property compression.type=gzip
{code}

The above code creates a file that is the same size as the default for {{max.request.size}} and the added overhead of the message pushes the uncompressed size over the limit.  Compressing the message ahead of time allows the message to go through.  When the message is blocked, the following exception is produced:
{code}
[2016-09-14 08:56:19,558] ERROR Error when sending message to topic tester with key: null, value: 1048576 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The message is 1048610 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
{code}

For completeness, I have confirmed that the console producer is setting {{compression.type}} properly by enabling DEBUG so this appears to be a problem in the size estimate of the message itself.  I would suggest we compress before we serialize instead of the other way around to avoid this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)