How to decommission a broker so the controller doesn't return it in the list of known brokers?

[Jeff Widman <je...@jeffwidman.com>]

How do I permanently remove a broker from a Kafka cluster?

Scenario:

I have a stable cluster of 3 brokers. I temporarily added a fourth broker
that successfully joined the cluster. The controller returned metadata
indicating this broker was part of the cluster.

However, I never rebalanced partitions onto this broker, so this broker #4
was never actually used, didn't join any replica lists, ISRs, etc.

I later decided to remove this unused broker from the cluster. I shutdown
the broker successfully and Zookeeper /broker/ids no longer lists broker #4.

However, when my application code connects to any Kafka broker and fetches
metadata, I get a broker list that includes this deleted broker.

How do I indicate to the cluster that this broker has been permanently
removed from the cluster and not just a transient downtime?

Additionally, what's happening under the covers that causes this?

I'm guessing that when I connect to a broker and ask for metadata, the
broker checks its local cache for the controller ID, contacts the broker
and asks it for the list of all brokers. Then the controller checks it's
cached list of brokers and returns the list of all brokers known to have
belonged to the cluster at any point in time.

I'm guessing this happens because it's not certain if the dead broker is
permanently removed or just transient downtime. So I'm thinking I just need
to indicate to the controller that it needs to reset it's list of known
cluster brokers to the known live brokers in zookeeper. But would not be
surprised if something in my mental model is incorrect.

This is for Kafka 0.8.2. I am planning to upgrade to 0.10 in the
not-to-distant future, so if 0.10 handles this differently, I'm also
curious about that.


-- 

*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><

Re: How to decommission a broker so the controller doesn't return it in the list of known brokers?

[Jeff Widman <je...@jeffwidman.com>]

It looks like this problem is caused by this bug in Kafka 8, which was
fixed in Kafka 9:

https://issues.apache.org/jira/browse/KAFKA-972


On Thu, Sep 8, 2016 at 3:55 PM, Jeff Widman <je...@jeffwidman.com> wrote:

> How do I permanently remove a broker from a Kafka cluster?
>
> Scenario:
>
> I have a stable cluster of 3 brokers. I temporarily added a fourth broker
> that successfully joined the cluster. The controller returned metadata
> indicating this broker was part of the cluster.
>
> However, I never rebalanced partitions onto this broker, so this broker #4
> was never actually used, didn't join any replica lists, ISRs, etc.
>
> I later decided to remove this unused broker from the cluster. I shutdown
> the broker successfully and Zookeeper /broker/ids no longer lists broker #4.
>
> However, when my application code connects to any Kafka broker and fetches
> metadata, I get a broker list that includes this deleted broker.
>
> How do I indicate to the cluster that this broker has been permanently
> removed from the cluster and not just a transient downtime?
>
> Additionally, what's happening under the covers that causes this?
>
> I'm guessing that when I connect to a broker and ask for metadata, the
> broker checks its local cache for the controller ID, contacts the broker
> and asks it for the list of all brokers. Then the controller checks it's
> cached list of brokers and returns the list of all brokers known to have
> belonged to the cluster at any point in time.
>
> I'm guessing this happens because it's not certain if the dead broker is
> permanently removed or just transient downtime. So I'm thinking I just need
> to indicate to the controller that it needs to reset it's list of known
> cluster brokers to the known live brokers in zookeeper. But would not be
> surprised if something in my mental model is incorrect.
>
> This is for Kafka 0.8.2. I am planning to upgrade to 0.10 in the
> not-to-distant future, so if 0.10 handles this differently, I'm also
> curious about that.
>
>
> --
>
> *Jeff Widman*
> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> <><
>



-- 

*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><