[jira] [Created] (KAFKA-8812) Rebalance Producers - yes, I mean it ;-)

["Werner Daehn (JIRA)" <ji...@apache.org>]

Werner Daehn created KAFKA-8812:
-----------------------------------

             Summary: Rebalance Producers - yes, I mean it ;-)
                 Key: KAFKA-8812
                 URL: https://issues.apache.org/jira/browse/KAFKA-8812
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 2.3.0
            Reporter: Werner Daehn


Please bare with me. Initially this thought sounds stupid but it has its merits.

 

How do you build a distributed producer at the moment? You use Kafka Connect which in turn requires a cluster that tells which instance is producing what partitions.

On the consumer side it is different. There Kafka itself does the data distribution. If you have 10 Kafka partitions and 10 consumers, each will get data for one partition. With 5 consumers, each will get data from two partitions. And if there is only a single consumer active, it gets all data. All is managed by Kafka, all you have to do is start as many consumers as you want.

 

I'd like to suggest something similar for the producers. A producer would tell Kafka that its source has 10 partitions. The Kafka server then responds with a list of partitions this instance shall be responsible for. If it is the only producer, the response would be all 10 partitions. If it is the second instance starting up, the first instance would get the information it should produce data for partition 1-5 and the new one for partition 6-10. If the producer fails to respond with an alive packet, a rebalance does happen, informing the active producer to take more load and the dead producer will get an error when sending data again.

For restart, the producer rebalance has to send the starting point where to start producing the data onwards from as well, of course. Would be best if this is a user generated pointer and not the topic offset. Then it can be e.g. the database system change number, a database transaction id or something similar.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)