Producer/Consumer config for length Kafka broker maintenance

[Harald Kirsch <ha...@raytion.com>]

Hi all,

there are so many timeouts to tweak mentioned in the documentation that 
I wonder what the correct configuration for producer and consumer is to 
survive a, say, 1 hour, broker shutdown.

With "survive" I mean that the processes are idle or blocked and keep 
trying to send their data, and just pick up work shortly after the 
broker appears again.

I have the following suspects to tweak:

PRODUCER:
connections.max.idle.ms: (9 minutes) Does a lost connection count as idle?

max.block.ms: (default 1 minute) Seems a definite candidate to raise to 
1 hour.

request.timeout.ms: (default 1/2 minute)
metadata.fetch.timeout.ms: (default 1 minute)
retry.backoff.ms: (default 100ms)

CONSUMER:
connections.max.idle.ms: (same as above I guess)
request.timeout.ms

What would be a good combination of settings?

Harald.

Re: Producer/Consumer config for length Kafka broker maintenance

[Tom Crayford <tc...@heroku.com>]

Hi there,

If you're only shutting down a single broker of many and you have a
replication factor more than 1, those consumer and producer configs should
handle it. However, if you have only a single broker, I'd recommend getting
some replication in before doing any maintenance work - Kafka is really
designed to operate with replication.

Thanks

Tom Crayford
Heroku Kafka

On Tue, Sep 6, 2016 at 5:30 AM, Harald Kirsch <ha...@raytion.com>
wrote:

> Hi all,
>
> there are so many timeouts to tweak mentioned in the documentation that I
> wonder what the correct configuration for producer and consumer is to
> survive a, say, 1 hour, broker shutdown.
>
> With "survive" I mean that the processes are idle or blocked and keep
> trying to send their data, and just pick up work shortly after the broker
> appears again.
>
> I have the following suspects to tweak:
>
> PRODUCER:
> connections.max.idle.ms: (9 minutes) Does a lost connection count as idle?
>
> max.block.ms: (default 1 minute) Seems a definite candidate to raise to 1
> hour.
>
> request.timeout.ms: (default 1/2 minute)
> metadata.fetch.timeout.ms: (default 1 minute)
> retry.backoff.ms: (default 100ms)
>
> CONSUMER:
> connections.max.idle.ms: (same as above I guess)
> request.timeout.ms
>
> What would be a good combination of settings?
>
> Harald.
>