To what value should SpoutConf.fetchSizeBytes be set?

[François Méthot <fm...@gmail.com>]

Hi,

  We have been having issue with reading from our Topic. The topic would
stop reading from the queue without warning or error. It would be stuck at
a specific offset of a specific partition.

  After a lot of debugging and testing, we found out that by increasing the
SpoutConfig.fetchSizeBytes from the default 1MB to 2MB or more, the problem
would go away. It would be able to consume the offset where it used to stop.

To what value should fetchSizeBytes be set?

Should we have seen a warning/error in the log when the spout can`t iterate
through message within the read bytebuffer?

In kafka, we have message.max.bytes and replica.fetch.max.bytes set to
10MB, is there a link between those setting and the fetchSizeBytes.

Thanks a lot
François

Re: To what value should SpoutConf.fetchSizeBytes be set?

[Harsha <st...@harsha.io>]

fetchSizeBytes is equivalent kafka consumer’s fetch.message.max.bytes . This should be at a minimum equal to server’s max.message.bytes or more. In your case it should be at least 10MB. Also you need to increase buffersizebytes as well.
-- 
Harsha


On March 26, 2015 at 8:31:39 AM, François Méthot (fmethot78@gmail.com) wrote:

 
Hi,
 
  We have been having issue with reading from our Topic. The topic would stop reading from the queue without warning or error. It would be stuck at a specific offset of a specific partition.
 
  After a lot of debugging and testing, we found out that by increasing the SpoutConfig.fetchSizeBytes from the default 1MB to 2MB or more, the problem would go away. It would be able to consume the offset where it used to stop.
 
To what value should fetchSizeBytes be set?
 
Should we have seen a warning/error in the log when the spout can`t iterate through message within the read bytebuffer?
 
In kafka, we have message.max.bytes and replica.fetch.max.bytes set to 10MB, is there a link between those setting and the fetchSizeBytes.
 
Thanks a lot
François