Re: Allow keys to specify partitionKey

[Boyang Chen <re...@gmail.com>]

Hey Jan,

correct me if I'm wrong, my understanding of your problem is as follows:

1. This is a Kafka Streams question
2. You want to preserve the same primary key, but to repartition based on
some certain field with key/value

If this is the case, have you considered using `through(topic, Produced<>)`
interface to inject some customized partitioner?


On Mon, Oct 21, 2019 at 8:05 AM Jan Bols <gu...@gmail.com> wrote:

> The default partitioner takes a hash of the key of a topic to determine
> the partition number.
>
> It would be useful for a key to be able to specify the object on which the
> default partitioner should base its hash on. This would allow us to use
> different composite keys and still be certain that they arrive on the same
> partition.
>
> For example: I have 2 topics. One keys with a composite key containing a
> student with its classRoom. The other one is keyed with a treacher and its
> classRoom.
>
> I want to make sure both end up in the same partition, even though they
> have different keys. However, they share the same classRoom.
>
> I can create a custom partitioner where I specifically partition by
> classRoom, but things break down when using kafka-streams because
> kafka-streams always uses the default partitioner for internal
> repartitioning during maps, joins, etc....
>
> Would it be a useful idea to allow keys to implement a well-known
> interface like the following:
> interface PartitionKeyAware{
>   Object partitionKey()
> }
>
> The default partitioner could then look for objects of that type and call
> the partitionKey to do its hash.
>
> Or is this a really bad idea not worth the effort to put it in a KIP?
>
>