Compressed message size limits

[Anishek Agarwal <an...@gmail.com>]

Hello,

i am trying to send compressed messages to kafka. topic/broker
configurations are default, i have provided "compression.type" of "snappy"
on kafka producer. the uncompressed message size is 1160668 bytes, error i
get is

*org.apache.kafka.common.errors.RecordTooLargeException: The message is
1160694 bytes when serialized which is larger than the maximum request size
you have configured with the max.request.size configuration.*

difference i am assuming is the header etc

i am using org.apache.kafka:kafka-clients:0.8.2.1

I would think the message size validation would be done after the message
is compressed ?

i looked at the source code and its done on the raw serialized value and
not the compressed value ? is this correct behavior ?  its on tag 0.8.2.1 -
KafkaProducer.java (line 335)

seems to be related @ https://issues.apache.org/jira/browse/KAFKA-1718

Please do let me know if i am interpreting this wrong?

I also looked at the kafka-python code a bit and seems that it does not do
the validation before sending messages and it creates the message set using
the codec specified

thanks
anishek