Prevent broker from leading topic partitions

[Jason Aliyetti <j....@gmail.com>]

I have a use case that requires a 2 node deployment with a Kafka-backed
service with the following constraints:

- All data must be persisted to node 1.  If node 1 fails (regardless of the
status of node 2), then the system must stop.
- If node 2 is up, then it must stay in synch with node 1.
- If node 2 fails, then service must not be disrupted, but as soon as it
comes back up and rejoins ISR it must stay in synch.

The deployment is basically a primary node and a cold node with real time
replication, but no failover to the cold node.

To achieve this I am considering adding a broker-level configuration option
that would prevent a broker from becoming a leader for any topic partition
it hosts - this would allow me to enforce that the cold node never take
leadership for any topics.  In conjunction with manipulating a topic's
"min.insync.replicas" setting at runtime, I should be able to achieve the
behavior desired (2 if both brokers up, 1 if the standby goes down).


I know this sounds like an edgy use case, but does this sound like a
reasonable approach?  Are there any valid use cases around such a broker or
topic level configuration (i.e. does this sound like a feature that would
make sense to open a KIP against)?

Re: Prevent broker from leading topic partitions

[Alexey Ozeritsky <ao...@yandex.ru>]

Hi

22.08.2016, 19:16, "Tom Crayford" <tc...@heroku.com>:
> Hi,
>
> I don't think I understand *why* you need this. Kafka is by default a
> distributed HA system that balances data and leadership over nodes. Why do
> you need to change this?

In fact, kafka does not balance anything. It uses static partition distribution.
In most cases the first replica in replica list is always a leader.

If a replica is under heavy load (for example hard disk was replaced or so on) it is a bad idea to do automatic leader "rebalance".

>
> You could accomplish something like this with mirror maker, that may make
> more sense.
>
> Thanks
>
> Tom Crayford
>
> Heroku Kafka
>
> On Mon, Aug 22, 2016 at 4:05 PM, Jason Aliyetti <j....@gmail.com>
> wrote:
>
>> �I have a use case that requires a 2 node deployment with a Kafka-backed
>> �service with the following constraints:
>>
>> �- All data must be persisted to node 1. If node 1 fails (regardless of the
>> �status of node 2), then the system must stop.
>> �- If node 2 is up, then it must stay in synch with node 1.
>> �- If node 2 fails, then service must not be disrupted, but as soon as it
>> �comes back up and rejoins ISR it must stay in synch.
>>
>> �The deployment is basically a primary node and a cold node with real time
>> �replication, but no failover to the cold node.
>>
>> �To achieve this I am considering adding a broker-level configuration option
>> �that would prevent a broker from becoming a leader for any topic partition
>> �it hosts - this would allow me to enforce that the cold node never take
>> �leadership for any topics. In conjunction with manipulating a topic's
>> �"min.insync.replicas" setting at runtime, I should be able to achieve the
>> �behavior desired (2 if both brokers up, 1 if the standby goes down).
>>
>> �I know this sounds like an edgy use case, but does this sound like a
>> �reasonable approach? Are there any valid use cases around such a broker or
>> �topic level configuration (i.e. does this sound like a feature that would
>> �make sense to open a KIP against)?

Re: Prevent broker from leading topic partitions

[Tom Crayford <tc...@heroku.com>]

Hi,

I don't think I understand *why* you need this. Kafka is by default a
distributed HA system that balances data and leadership over nodes. Why do
you need to change this?

You could accomplish something like this with mirror maker, that may make
more sense.

Thanks

Tom Crayford

Heroku Kafka

On Mon, Aug 22, 2016 at 4:05 PM, Jason Aliyetti <j....@gmail.com>
wrote:

> I have a use case that requires a 2 node deployment with a Kafka-backed
> service with the following constraints:
>
> - All data must be persisted to node 1.  If node 1 fails (regardless of the
> status of node 2), then the system must stop.
> - If node 2 is up, then it must stay in synch with node 1.
> - If node 2 fails, then service must not be disrupted, but as soon as it
> comes back up and rejoins ISR it must stay in synch.
>
> The deployment is basically a primary node and a cold node with real time
> replication, but no failover to the cold node.
>
> To achieve this I am considering adding a broker-level configuration option
> that would prevent a broker from becoming a leader for any topic partition
> it hosts - this would allow me to enforce that the cold node never take
> leadership for any topics.  In conjunction with manipulating a topic's
> "min.insync.replicas" setting at runtime, I should be able to achieve the
> behavior desired (2 if both brokers up, 1 if the standby goes down).
>
>
> I know this sounds like an edgy use case, but does this sound like a
> reasonable approach?  Are there any valid use cases around such a broker or
> topic level configuration (i.e. does this sound like a feature that would
> make sense to open a KIP against)?
>