Re: Max message size and compression

[Krzysztof Zarzycki <k....@gmail.com>]

Hi all,
As far as I understand this Jira
https://issues.apache.org/jira/browse/KAFKA-4169, unfortunately
max.request.size works on *uncompressed* messages. Which means producer
won't be able to send messages larger than that limit, no matter
compression. The fragment above is still right though, because it considers
only broker side.

I'm thinking if it was possible (but somewhat risky) to define
max.request.size on producer side larger (e.g. 2MB) to account the typical
compression ratio of the data (knowing it apriori). Will that work?


Thanks,
Krzysztof


2017-06-22 15:55 GMT+02:00 mayank rathi <ma...@gmail.com>:

> Hello Eli,
>
> This is from Kafka: Definitive Guide ( by Neha Narkhede , Gwen Shapira ,
> and Todd Palino) , Chapter 2. Installing Kafka
>
> "The Kafka broker limits the maximum size of a message that can be
> produced, configured by the message.max.bytes parameter which defaults to
> 1000000, or 1 megabyte. A producer which tries to send a message larger
> than this will receive an error back from the broker and the message will
> not be accepted. As with all byte sizes specified on the broker, this
> configuration deals with compressed message size, which means that
> producers can send messages that are much larger than this value
> uncompressed, provided they compress down to under the configured
> message.max.bytes size."
>
> Thanks!!
>
> On Thu, Jun 22, 2017 at 4:18 AM, Eli Jordan <el...@gmail.com>
> wrote:
>
> > Thanks for the reply Mayank. Do you know if this is documented somewhere?
> > I wasnt able to find mention of it.
> >
> > Thanks
> > Eli
> >
> > > On 22 Jun 2017, at 05:50, mayank rathi <ma...@gmail.com> wrote:
> > >
> > > If you are compressing messages than size of "compressed" message
> should
> > be
> > > less than what's specified in these parameters.
> > >
> > > On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan <el...@gmail.com>
> > > wrote:
> > >
> > >> Hi
> > >>
> > >> max.message.bytes controls the maximum message size the kafka server
> > will
> > >> process
> > >>
> > >> message.max.bytes controls the maximum message size the consumer will
> > >> process
> > >>
> > >> max.request.size controls the maximum request size for the producer
> > >>
> > >> Whats not clear to me (and I can't find documented anywhere) is if the
> > >> message size limits are imposed on compressed or uncompressed
> messages,
> > >> when compression is enabled.
> > >>
> > >> Note: I'm use Kafka 0.10.2.0 if that makes any difference.
> > >>
> > >> Any pointers or advice on this would be greatly appreciated.
> > >>
> > >> Thanks
> > >> Eli
> > >
> > >
> > >
> > >
> > > --
> > > NOTICE: This email message is for the sole use of the intended
> > recipient(s)
> > > and may contain confidential and privileged information. Any
> unauthorized
> > > review, use, disclosure or distribution is prohibited. If you are not
> the
> > > intended recipient, please contact the sender by reply email and
> destroy
> > > all copies of the original message.
> >
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended recipient(s)
> and may contain confidential and privileged information. Any unauthorized
> review, use, disclosure or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy
> all copies of the original message.
>

Re: Max message size and compression

[Matt Farmer <ma...@frmr.me>]

I can confirm that the message size check in the producer works on the
uncompressed size as of 0.11.0.1, as I had to investigate this internally.
:)

I've got a similar problem with messages that can occasionally exceed this
limit. We're taking the approach of enforcing a hard size limit when events
are consumed at the start of our processing pipeline and setting a "soft
limit" of higher than that as our max.request.size for all subsequent
applications on that pipeline.

I think you will need to make sure that your brokers are configured to
replicate messages that large. I think if you push a single message that's
2MB, but the brokers have limits on their replication size you could run
into issues? Could be wrong though. Working from a vague memory of broker
configuration at the moment.

On Wed, Oct 18, 2017 at 11:20 AM Krzysztof Zarzycki <k....@gmail.com>
wrote:

> Hi all,
> As far as I understand this Jira
> https://issues.apache.org/jira/browse/KAFKA-4169, unfortunately
> max.request.size works on *uncompressed* messages. Which means producer
> won't be able to send messages larger than that limit, no matter
> compression. The fragment above is still right though, because it considers
> only broker side.
>
> I'm thinking if it was possible (but somewhat risky) to define
> max.request.size on producer side larger (e.g. 2MB) to account the typical
> compression ratio of the data (knowing it apriori). Will that work?
>
>
> Thanks,
> Krzysztof
>
>
> 2017-06-22 15:55 GMT+02:00 mayank rathi <ma...@gmail.com>:
>
> > Hello Eli,
> >
> > This is from Kafka: Definitive Guide ( by Neha Narkhede , Gwen Shapira ,
> > and Todd Palino) , Chapter 2. Installing Kafka
> >
> > "The Kafka broker limits the maximum size of a message that can be
> > produced, configured by the message.max.bytes parameter which defaults to
> > 1000000, or 1 megabyte. A producer which tries to send a message larger
> > than this will receive an error back from the broker and the message will
> > not be accepted. As with all byte sizes specified on the broker, this
> > configuration deals with compressed message size, which means that
> > producers can send messages that are much larger than this value
> > uncompressed, provided they compress down to under the configured
> > message.max.bytes size."
> >
> > Thanks!!
> >
> > On Thu, Jun 22, 2017 at 4:18 AM, Eli Jordan <el...@gmail.com>
> > wrote:
> >
> > > Thanks for the reply Mayank. Do you know if this is documented
> somewhere?
> > > I wasnt able to find mention of it.
> > >
> > > Thanks
> > > Eli
> > >
> > > > On 22 Jun 2017, at 05:50, mayank rathi <ma...@gmail.com>
> wrote:
> > > >
> > > > If you are compressing messages than size of "compressed" message
> > should
> > > be
> > > > less than what's specified in these parameters.
> > > >
> > > > On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan <
> elias.k.jordan@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> max.message.bytes controls the maximum message size the kafka server
> > > will
> > > >> process
> > > >>
> > > >> message.max.bytes controls the maximum message size the consumer
> will
> > > >> process
> > > >>
> > > >> max.request.size controls the maximum request size for the producer
> > > >>
> > > >> Whats not clear to me (and I can't find documented anywhere) is if
> the
> > > >> message size limits are imposed on compressed or uncompressed
> > messages,
> > > >> when compression is enabled.
> > > >>
> > > >> Note: I'm use Kafka 0.10.2.0 if that makes any difference.
> > > >>
> > > >> Any pointers or advice on this would be greatly appreciated.
> > > >>
> > > >> Thanks
> > > >> Eli
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > NOTICE: This email message is for the sole use of the intended
> > > recipient(s)
> > > > and may contain confidential and privileged information. Any
> > unauthorized
> > > > review, use, disclosure or distribution is prohibited. If you are not
> > the
> > > > intended recipient, please contact the sender by reply email and
> > destroy
> > > > all copies of the original message.
> > >
> >
> >
> >
> > --
> > NOTICE: This email message is for the sole use of the intended
> recipient(s)
> > and may contain confidential and privileged information. Any unauthorized
> > review, use, disclosure or distribution is prohibited. If you are not the
> > intended recipient, please contact the sender by reply email and destroy
> > all copies of the original message.
> >
>