You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Igor Kravzov <ig...@gmail.com> on 2016/10/13 20:22:54 UTC

GetKafka maximum fetch size

Hi,

I am getting the following exception in nifi-0.6.1:

kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer. Increase the fetch size, or decrease
the maximum message size the broker will allow.

What is the max size? How can I increase max size?

Thanks in advance.

Re: GetKafka maximum fetch size

Posted by Igor Kravzov <ig...@gmail.com>.

Thanks Jeremy. Where can I set/change this property? in GetKafka?

On Thu, Oct 13, 2016 at 5:33 PM, Jeremy Farbota <jf...@payoff.com> wrote:

> Igor,
>
> Kafka consumer properties can be found here: http://kafka.apache.org/
> documentation.html#consumerconfigs
>
> GetKafka uses the old consumer so the consumer property is:
> fetch.message.max.bytes
>
> The default for that property is ~1M.
>
> If possible, you should limit the replica.fetch.max.bytes on the broker
> configs to avoid this error (reset it to the default of 1048576).
>
> AFAIK, you'll have to create a custom version of GetKafka that enables
> adjustments to fetch.message.max.bytes.
>
>
>
> On Thu, Oct 13, 2016 at 1:22 PM, Igor Kravzov <ig...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am getting the following exception in nifi-0.6.1:
>>
>> kafka.common.MessageSizeTooLargeException: Found a message larger than
>> the maximum fetch size of this consumer. Increase the fetch size, or
>> decrease the maximum message size the broker will allow.
>>
>> What is the max size? How can I increase max size?
>>
>> Thanks in advance.
>>
>
>
>
> --
>
> [image: Payoff, Inc.] <http://www.payoff.com/>
>
> Jeremy Farbota
> Software Engineer, Data
> jfarbota@payoff.com <em...@payoff.com> • (217) 898-8110 <(949)+430-0630>
>
> I'm a Storyteller. Discover your Financial Personality!
> <https://www.payoff.com/quiz>
>
> [image: Facebook]  <https://www.facebook.com/payoff> [image: Twitter]
> <https://www.twitter.com/payoff> [image: Linkedin]
> <https://www.linkedin.com/company/payoff-com>
>

Re: GetKafka maximum fetch size

Posted by Jeremy Farbota <jf...@payoff.com>.

Igor,

Kafka consumer properties can be found here:
http://kafka.apache.org/documentation.html#consumerconfigs

GetKafka uses the old consumer so the consumer property is:
fetch.message.max.bytes

The default for that property is ~1M.

If possible, you should limit the replica.fetch.max.bytes on the broker
configs to avoid this error (reset it to the default of 1048576).

AFAIK, you'll have to create a custom version of GetKafka that enables
adjustments to fetch.message.max.bytes.

On Thu, Oct 13, 2016 at 1:22 PM, Igor Kravzov <ig...@gmail.com>
wrote:

> Hi,
>
> I am getting the following exception in nifi-0.6.1:
>
> kafka.common.MessageSizeTooLargeException: Found a message larger than
> the maximum fetch size of this consumer. Increase the fetch size, or
> decrease the maximum message size the broker will allow.
>
> What is the max size? How can I increase max size?
>
> Thanks in advance.
>

-- 

[image: Payoff, Inc.] <http://www.payoff.com/>

Jeremy Farbota
Software Engineer, Data
jfarbota@payoff.com <em...@payoff.com> • (217) 898-8110 <(949)+430-0630>

I'm a Storyteller. Discover your Financial Personality!
<https://www.payoff.com/quiz>

[image: Facebook]  <https://www.facebook.com/payoff> [image: Twitter]
<https://www.twitter.com/payoff> [image: Linkedin]
<https://www.linkedin.com/company/payoff-com>