You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jun Rao <ju...@gmail.com> on 2013/08/01 07:03:07 UTC
Re: Consumer behavior when message exceeds fetch.message.max.bytes
If you look at ConsumerIterator, we will throw MessageSizeTooLargeException
to the caller if the fetch size is too small.
Thanks,
Jun
On Wed, Jul 31, 2013 at 8:25 AM, Sam Meder <sa...@jivesoftware.com>wrote:
> We're expecting to occasionally have to deal with pretty large messages
> being sent to Kafka. We will of course set the fetch size appropriately
> high, but are concerned about the behavior when the message exceeds the
> fetch size. As far as I can tell the current behavior when a message that
> is too large is encountered is to pretend it is not there and not notify
> the consumer in any way. IMO it would be better to throw an exception than
> silently ignoring the issue (with the current code one can't really
> distinguish a large message from no data at all).
>
> Thoughts?
>
> /Sam
Re: Consumer behavior when message exceeds fetch.message.max.bytes
Posted by Sam Meder <sa...@jivesoftware.com>.
We put together a test case that showws the exception is never thrown. I looked into it a bit more and filed https://issues.apache.org/jira/browse/KAFKA-994 which has the details.
/Sam
On Aug 1, 2013, at 7:03 AM, Jun Rao <ju...@gmail.com> wrote:
> If you look at ConsumerIterator, we will throw MessageSizeTooLargeException
> to the caller if the fetch size is too small.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jul 31, 2013 at 8:25 AM, Sam Meder <sa...@jivesoftware.com>wrote:
>
>> We're expecting to occasionally have to deal with pretty large messages
>> being sent to Kafka. We will of course set the fetch size appropriately
>> high, but are concerned about the behavior when the message exceeds the
>> fetch size. As far as I can tell the current behavior when a message that
>> is too large is encountered is to pretend it is not there and not notify
>> the consumer in any way. IMO it would be better to throw an exception than
>> silently ignoring the issue (with the current code one can't really
>> distinguish a large message from no data at all).
>>
>> Thoughts?
>>
>> /Sam