You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kostas Christidis <ko...@gmail.com> on 2017/03/23 03:26:37 UTC
Relationship fetch.replica.max.bytes and message.max.bytes
Can fetch.replica.max.bytes be equal to message.max.bytes?
1. The defaults in the official Kafka documentation [1] have the
parameter "fetch.replica.max.bytes" set to a higher value than
"message.max.bytes". However, nothing in the description of these
parameters implies that equality would be wrong.
2. The relevant passage in pg. 41 in the Definitive Guide book [2]
does not imply that the former needs to be larger than the latter
either.
3. A Cloudera doc [3] however notes that: "replica.fetch.max.bytes
[...] must be larger than message.max.bytes, or a broker can accept
messages it cannot replicate, potentially resulting in data loss."
4. The only other reference I could find to this strict inequality was
this StackOverflow comment [4].
So:
Does fetch.replica.max.bytes *have* to be strictly larger to message.max.bytes?
If so, what is the technical reason behind this?
Thank you.
[1] https://kafka.apache.org/documentation/
[2] https://shop.oreilly.com/product/0636920044123.do
[3] https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html
[4] http://stackoverflow.com/a/39026744/2363529
Re: Relationship fetch.replica.max.bytes and message.max.bytes
Posted by Kostas Christidis <ko...@gmail.com>.
Cool - thanks for clarifying this!
On Thu, Mar 23, 2017 at 10:54 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Kostas,
>
> Yes, equal is fine. The code that prints an error if replication fails due
> to this:
>
> error(s"Replication is failing due to a message that is greater than
> replica.fetch.max.bytes for partition $topicPartition. " +
> "This generally occurs when the max.message.bytes has been
> overridden to exceed this value and a suitably large " +
> "message has also been sent. To fix this problem increase
> replica.fetch.max.bytes in your broker config to be " +
> "equal or larger than your settings for max.message.bytes, both at
> a broker and topic level.")
>
> Ismael
>
> On Thu, Mar 23, 2017 at 2:37 PM, Kostas Christidis <ko...@gmail.com> wrote:
>
>> On Thu, Mar 23, 2017 at 5:04 AM, Ben Stopford <be...@confluent.io> wrote:
>> > Hi Kostas - The docs for replica.fetch.max.bytes should be helpful here:
>> >
>> > The number of bytes of messages to attempt to fetch for each partition.
>> > This is not an absolute maximum, if the first message in the first
>> > non-empty partition of the fetch is larger than this value, the message
>> > will still be returned to ensure that progress can be made.
>>
>> Hi Ben
>>
>> When I'm reading this bit (which I have done before), I am left with
>> the impression that replica.fetch.max.bytes can in fact be equal to
>> message.max.bytes. Am I wrong? This interpretation goes against points
>> 3 and 4 in the original email.
>>
>> Kostas
>>
Re: Relationship fetch.replica.max.bytes and message.max.bytes
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Kostas,
Yes, equal is fine. The code that prints an error if replication fails due
to this:
error(s"Replication is failing due to a message that is greater than
replica.fetch.max.bytes for partition $topicPartition. " +
"This generally occurs when the max.message.bytes has been
overridden to exceed this value and a suitably large " +
"message has also been sent. To fix this problem increase
replica.fetch.max.bytes in your broker config to be " +
"equal or larger than your settings for max.message.bytes, both at
a broker and topic level.")
Ismael
On Thu, Mar 23, 2017 at 2:37 PM, Kostas Christidis <ko...@gmail.com> wrote:
> On Thu, Mar 23, 2017 at 5:04 AM, Ben Stopford <be...@confluent.io> wrote:
> > Hi Kostas - The docs for replica.fetch.max.bytes should be helpful here:
> >
> > The number of bytes of messages to attempt to fetch for each partition.
> > This is not an absolute maximum, if the first message in the first
> > non-empty partition of the fetch is larger than this value, the message
> > will still be returned to ensure that progress can be made.
>
> Hi Ben
>
> When I'm reading this bit (which I have done before), I am left with
> the impression that replica.fetch.max.bytes can in fact be equal to
> message.max.bytes. Am I wrong? This interpretation goes against points
> 3 and 4 in the original email.
>
> Kostas
>
Re: Relationship fetch.replica.max.bytes and message.max.bytes
Posted by Kostas Christidis <ko...@gmail.com>.
On Thu, Mar 23, 2017 at 5:04 AM, Ben Stopford <be...@confluent.io> wrote:
> Hi Kostas - The docs for replica.fetch.max.bytes should be helpful here:
>
> The number of bytes of messages to attempt to fetch for each partition.
> This is not an absolute maximum, if the first message in the first
> non-empty partition of the fetch is larger than this value, the message
> will still be returned to ensure that progress can be made.
Hi Ben
When I'm reading this bit (which I have done before), I am left with
the impression that replica.fetch.max.bytes can in fact be equal to
message.max.bytes. Am I wrong? This interpretation goes against points
3 and 4 in the original email.
Kostas
Re: Relationship fetch.replica.max.bytes and message.max.bytes
Posted by Manikumar <ma...@gmail.com>.
Also, even though replica fetcher makes progress, individual message size
should be less than
or equal to message.max.bytes. otherwise, we will get
RecordTooLargeException.
On Thu, Mar 23, 2017 at 2:34 PM, Ben Stopford <be...@confluent.io> wrote:
> Hi Kostas - The docs for replica.fetch.max.bytes should be helpful here:
>
> The number of bytes of messages to attempt to fetch for each partition.
> This is not an absolute maximum, if the first message in the first
> non-empty partition of the fetch is larger than this value, the message
> will still be returned to ensure that progress can be made.
>
> -B
>
> On Thu, Mar 23, 2017 at 3:27 AM Kostas Christidis <ko...@gmail.com>
> wrote:
>
> > Can fetch.replica.max.bytes be equal to message.max.bytes?
> >
> > 1. The defaults in the official Kafka documentation [1] have the
> > parameter "fetch.replica.max.bytes" set to a higher value than
> > "message.max.bytes". However, nothing in the description of these
> > parameters implies that equality would be wrong.
> >
> > 2. The relevant passage in pg. 41 in the Definitive Guide book [2]
> > does not imply that the former needs to be larger than the latter
> > either.
> >
> > 3. A Cloudera doc [3] however notes that: "replica.fetch.max.bytes
> > [...] must be larger than message.max.bytes, or a broker can accept
> > messages it cannot replicate, potentially resulting in data loss."
> >
> > 4. The only other reference I could find to this strict inequality was
> > this StackOverflow comment [4].
> >
> > So:
> >
> > Does fetch.replica.max.bytes *have* to be strictly larger to
> > message.max.bytes?
> >
> > If so, what is the technical reason behind this?
> >
> > Thank you.
> >
> > [1] https://kafka.apache.org/documentation/
> > [2] https://shop.oreilly.com/product/0636920044123.do
> > [3]
> > https://www.cloudera.com/documentation/kafka/latest/
> topics/kafka_performance.html
> > [4] http://stackoverflow.com/a/39026744/2363529
> >
>
Re: Relationship fetch.replica.max.bytes and message.max.bytes
Posted by Ben Stopford <be...@confluent.io>.
Hi Kostas - The docs for replica.fetch.max.bytes should be helpful here:
The number of bytes of messages to attempt to fetch for each partition.
This is not an absolute maximum, if the first message in the first
non-empty partition of the fetch is larger than this value, the message
will still be returned to ensure that progress can be made.
-B
On Thu, Mar 23, 2017 at 3:27 AM Kostas Christidis <ko...@gmail.com> wrote:
> Can fetch.replica.max.bytes be equal to message.max.bytes?
>
> 1. The defaults in the official Kafka documentation [1] have the
> parameter "fetch.replica.max.bytes" set to a higher value than
> "message.max.bytes". However, nothing in the description of these
> parameters implies that equality would be wrong.
>
> 2. The relevant passage in pg. 41 in the Definitive Guide book [2]
> does not imply that the former needs to be larger than the latter
> either.
>
> 3. A Cloudera doc [3] however notes that: "replica.fetch.max.bytes
> [...] must be larger than message.max.bytes, or a broker can accept
> messages it cannot replicate, potentially resulting in data loss."
>
> 4. The only other reference I could find to this strict inequality was
> this StackOverflow comment [4].
>
> So:
>
> Does fetch.replica.max.bytes *have* to be strictly larger to
> message.max.bytes?
>
> If so, what is the technical reason behind this?
>
> Thank you.
>
> [1] https://kafka.apache.org/documentation/
> [2] https://shop.oreilly.com/product/0636920044123.do
> [3]
> https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html
> [4] http://stackoverflow.com/a/39026744/2363529
>