Managing broker rolls?

[Matt Farmer <ma...@frmr.me>]

Hey all,

So, I'm curious to hear how others have solved this problem.

We've got quite a few brokers and rolling all of them to pick up new
configuration (which consists of triggering a clean shutdown, then
restarting the service and waiting for replication to catch up before
moving on) ultimately takes an entire day to do as a human. This is a
process I would like to automate.

Things that I have looked at include:

(1) Using a bot that can talk to the Kafka admin API - but there's
currently no Admin API call to trigger a clean shutdown of a broker (would
folks be interested in this?)

(2) Using a giant shell script that speaks admin API and can detect ISR
catch-up — but this either requires a developer's machine to stay connected
during the entire process (not a guarantee) or requires us to give some
shared resource SSH permissions across all our servers (not ideal)

What are others doing?

Would folks be interested in an AdminClient call that triggers a graceful
shutdown on a Broker? I could write up a KIP for this if so.

Cheers,
Matt

Re: Managing broker rolls?

[Matt Farmer <ma...@frmr.me>]

Bump, still haven't seen anything here. Betting this problem isn't unique
to us. Would love to hear how other folks are managing controlled restarts
of their clusters. =)

On Tue, Nov 28, 2017 at 4:40 PM Matt Farmer <ma...@frmr.me> wrote:

> Hey all,
>
> So, I'm curious to hear how others have solved this problem.
>
> We've got quite a few brokers and rolling all of them to pick up new
> configuration (which consists of triggering a clean shutdown, then
> restarting the service and waiting for replication to catch up before
> moving on) ultimately takes an entire day to do as a human. This is a
> process I would like to automate.
>
> Things that I have looked at include:
>
> (1) Using a bot that can talk to the Kafka admin API - but there's
> currently no Admin API call to trigger a clean shutdown of a broker (would
> folks be interested in this?)
>
> (2) Using a giant shell script that speaks admin API and can detect ISR
> catch-up — but this either requires a developer's machine to stay connected
> during the entire process (not a guarantee) or requires us to give some
> shared resource SSH permissions across all our servers (not ideal)
>
> What are others doing?
>
> Would folks be interested in an AdminClient call that triggers a graceful
> shutdown on a Broker? I could write up a KIP for this if so.
>
> Cheers,
> Matt
>