max.message.size, 0.7 SyncProducer, and compression

[Philip O'Toole <ph...@loggly.com>]

http://kafka.apache.org/07/configuration.html

Hello -- I can look the at the code too, but how does this setting interact
with compression? After all, a Producer doing compression doesn't know the
size of a "message" on the wire it will send to a Kafka broker until after
it has been compressed. And after it has been compressed, what if it is
greater than the configured value?

Or perhaps it applies to the "message" pre-compression?

Thanks,

Philip

Re: max.message.size, 0.7 SyncProducer, and compression

[Neha Narkhede <ne...@gmail.com>]

If compression is turned on, the check is done on the compressed message
size. The producer knows the compressed message size before it writes it on
the network.

Thanks,
Neha


On Tue, Jan 28, 2014 at 7:42 AM, Philip O'Toole <ph...@loggly.com> wrote:

> http://kafka.apache.org/07/configuration.html
>
> Hello -- I can look the at the code too, but how does this setting interact
> with compression? After all, a Producer doing compression doesn't know the
> size of a "message" on the wire it will send to a Kafka broker until after
> it has been compressed. And after it has been compressed, what if it is
> greater than the configured value?
>
> Or perhaps it applies to the "message" pre-compression?
>
> Thanks,
>
> Philip
>

Re: max.message.size, 0.7 SyncProducer, and compression

[Neha Narkhede <ne...@gmail.com>]

If compression is turned on, this applies to the size of the compressed
message and the producer knows the size before it writes the compressed
message on the wire.

Thanks,
Neha
On Jan 28, 2014 7:42 AM, "Philip O'Toole" <ph...@loggly.com> wrote:

> http://kafka.apache.org/07/configuration.html
>
> Hello -- I can look the at the code too, but how does this setting interact
> with compression? After all, a Producer doing compression doesn't know the
> size of a "message" on the wire it will send to a Kafka broker until after
> it has been compressed. And after it has been compressed, what if it is
> greater than the configured value?
>
> Or perhaps it applies to the "message" pre-compression?
>
> Thanks,
>
> Philip
>