Increasing partition count and preserving local order for a key

["Auel, Tarek" <ta...@sap.com>]

Hi,

I would like to increase the number of a partition in a topic while preserving the local order within a key. I do know that if I have a simple hashing algorithm, say h(x) = x mod partitionCnt, the order is not preserved if I increase the number of partitions. My goal is that a consumer is able to consume messages for a specific key in-order.

I think something like this is not implemented in Kafka, but maybe there are already best practices patterns or some kind of library that implements this. I'd highly appreciate if you have some pointers for me.

Cheers
Tarek

Re: Increasing partition count and preserving local order for a key

[Ian Wrigley <ia...@confluent.io>]

Hi

You can’t move existing records between partitions, but one possibility would be to create a new topic with the required number of partitions, then copy the data from the original topic to the new one. The default partitioning algorithm would ensure that all records with the same key in the new topic would be in the same partition.

Ian.


> On Mar 17, 2017, at 1:38 AM, Auel, Tarek <ta...@sap.com> wrote:
> 
> Hi,
> 
> I would like to increase the number of a partition in a topic while preserving the local order within a key. I do know that if I have a simple hashing algorithm, say h(x) = x mod partitionCnt, the order is not preserved if I increase the number of partitions. My goal is that a consumer is able to consume messages for a specific key in-order.
> 
> I think something like this is not implemented in Kafka, but maybe there are already best practices patterns or some kind of library that implements this. I'd highly appreciate if you have some pointers for me.
> 
> Cheers
> Tarek