Kafka "long" batch settings

[Michel Betancourt <mi...@gmail.com>]

Hi Kafka Users.  Using kafka client and server of 0.11.0.2


I am trying trying to set a kafka consumer for long batching.  I have tired
small/short batches with settings like these and it seems to work just fine:


poll timeout = 2000


max.poll.interval.ms = 50000

fetch.max.wait.ms = 500

fetch.min.bytes = 1

max.partition.fetch.bytes = 104857600

max.poll.records = 100

request.timeout.ms = 305000


But when trying to set a longer batch window we see Kafka get stuck from
time to time, eg settings:

poll timeout = 800000


max.poll.interval.ms = 850000

fetch.max.wait.ms = 720000

fetch.min.bytes = 200000

max.partition.fetch.bytes = 104857600

max.poll.records = 100

request.timeout.ms = 900000


Any thoughts on whether a long 10-12m batch window is feasible and if so
are there both server and client settings to have to re-configure to ensure
stability?



Thoughts?


-- 
-Michel