You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by manish jaiswal <ma...@gmail.com> on 2016/01/12 08:25:23 UTC

flafka comsumer fetch size

i m trying to read more than 1mb msg from kafka using flume
and i m getting fetch size error.

where to define in flume config for fetch.message.max.bytes.




error:
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer on topic test partition 2 at fetch
offset 0. Increase the fetch size, or decrease the maximum message size the
broker will allow.
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91)
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at
org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306)
    at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
    at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
    at
org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374)
    at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
    at java.lang.Thread.run(Thread.java:745)

Re: flafka comsumer fetch size

Posted by Tim Williams <wi...@gmail.com>.
IIRC, just add it to your flume configs, eg. for a source:

tier1.sources.src1.kafka.fetch.message.max.bytes=

Thanks,
--tim

On Tue, Jan 12, 2016 at 7:25 AM, manish jaiswal <ma...@gmail.com> wrote:
> i m trying to read more than 1mb msg from kafka using flume
> and i m getting fetch size error.
>
> where to define in flume config for fetch.message.max.bytes.
>
>
>
>
> error:
> kafka.common.MessageSizeTooLargeException: Found a message larger than the
> maximum fetch size of this consumer on topic test partition 2 at fetch
> offset 0. Increase the fetch size, or decrease the maximum message size the
> broker will allow.
>     at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91)
>     at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
>     at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
>     at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
>     at
> org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306)
>     at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>     at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>     at
> org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374)
>     at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>     at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>     at java.lang.Thread.run(Thread.java:745)