How can we re-key and re-partition records based on that key

[Sachin Mittal <sj...@gmail.com>]

Hi,
I have a topic of records (K1, V1).
This topic is partitioned based on the K1.

I use multiple streams instances to process these records based on that key
so that each instance process certain partitions only.

Now I want to re-key the records in that topic with a format
(K2, V2) and push them into new topic partitioned based on that key (K2).

ie (K1, V1) <=> (K2, V2).

I wanted to know what would be the best way to doing such a transformation.?

What APIs I should look at doing the same?

Thanks
Sachin

Re: How can we re-key and re-partition records based on that key

["Matthias J. Sax" <ma...@confluent.io>]

If you only want to change the key, you can use #selectKey() -- if you
want to change key and value, you can use #map().

Stream will automatically repartition the data afterwards if required
(ie, if you do a group-by or join). If you want to force repartitioning,
you can just call #through() after you have set the new key.

(For version 0.10.0.x there is not auto repartioning and you need to
repartition manually via #through() for all cases.)


-Matthias

On 7/16/17 4:13 AM, Sachin Mittal wrote:
> Hi,
> I have a topic of records (K1, V1).
> This topic is partitioned based on the K1.
> 
> I use multiple streams instances to process these records based on that key
> so that each instance process certain partitions only.
> 
> Now I want to re-key the records in that topic with a format
> (K2, V2) and push them into new topic partitioned based on that key (K2).
> 
> ie (K1, V1) <=> (K2, V2).
> 
> I wanted to know what would be the best way to doing such a transformation.?
> 
> What APIs I should look at doing the same?
> 
> Thanks
> Sachin
>