You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by manish jaiswal <ma...@gmail.com> on 2016/01/12 08:25:23 UTC
flafka comsumer fetch size
i m trying to read more than 1mb msg from kafka using flume
and i m getting fetch size error.
where to define in flume config for fetch.message.max.bytes.
error:
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer on topic test partition 2 at fetch
offset 0. Increase the fetch size, or decrease the maximum message size the
broker will allow.
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at
org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306)
at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at
org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Re: flafka comsumer fetch size
Posted by Tim Williams <wi...@gmail.com>.
IIRC, just add it to your flume configs, eg. for a source:
tier1.sources.src1.kafka.fetch.message.max.bytes=
Thanks,
--tim
On Tue, Jan 12, 2016 at 7:25 AM, manish jaiswal <ma...@gmail.com> wrote:
> i m trying to read more than 1mb msg from kafka using flume
> and i m getting fetch size error.
>
> where to define in flume config for fetch.message.max.bytes.
>
>
>
>
> error:
> kafka.common.MessageSizeTooLargeException: Found a message larger than the
> maximum fetch size of this consumer on topic test partition 2 at fetch
> offset 0. Increase the fetch size, or decrease the maximum message size the
> broker will allow.
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
> at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at
> org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)