Consuming from N topics "co-partitioned"

[Francesco Guardiani <fr...@gmail.com>]

Hi everybody,
I wish to implement a KafkaConsumer that consumes messages from N
co-partitioned topics, in a similar way to KafkaStreams for join semantic.
e.g. for two consumers and two topics with two partitions:

C0 = {T0P0, T1P0}
C1 = {T0P1, T1P1}
...

Any ideas/tips?

I assume most of the work must be done at PartitionAssignor level, there is
any implementation of it that is able to assign partitions like this?

Thank you so much,
FG

-- 
Francesco Guardiani
Website: https://slinkydeveloper.com/
Twitter: https://twitter.com/SlinkyGuardiani

Github: https://github.com/slinkydeveloper

Re: Consuming from N topics "co-partitioned"

["Matthias J. Sax" <ma...@confluent.io>]

> I assume most of the work must be done at PartitionAssignor level, 

Correct.

> there is
>> any implementation of it that is able to assign partitions like this?

Well, you can look into `StreamsPartitionAssignor` (but it does much
more than just co-partitioning, hence, it might be hard to extract the
part of the logic you are interested in.

But even if you implement it from scratch, it should not be too difficult.

Our of curiosity: why not just use Kafka Streams?


-Matthias

On 1/8/20 9:00 AM, Francesco Guardiani wrote:
> Hi everybody,
> I wish to implement a KafkaConsumer that consumes messages from N
> co-partitioned topics, in a similar way to KafkaStreams for join semantic.
> e.g. for two consumers and two topics with two partitions:
> 
> C0 = {T0P0, T1P0}
> C1 = {T0P1, T1P1}
> ...
> 
> Any ideas/tips?
> 
> I assume most of the work must be done at PartitionAssignor level, there is
> any implementation of it that is able to assign partitions like this?
> 
> Thank you so much,
> FG
>