can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

[Pushkar Deole <pd...@gmail.com>]

Hi All,

We have a kafka streams application that consumes from 2 different topics
say topic A and topic B. The application uses data of telephone call on
those topics and each call has a call id which is used as key to send
events to those 2 topics. e.g. for a telephone call, the 1st event related
to that call is sent to A with call id however subsequent event for that
same call might go to topic B again with call id as key.

*At times, we need to process those 2 events in an order, which is not
possible with the current topology that we are using*. *Can someone suggest
if this is possible to achieve with streams?*
The topology is as below:

Topic A has 6 partitions
Topics B has 6 partitions
Call id used as key on both topics
Kafka streams application has 3 instances that consumes from both of the
topics as source topics.
Each streams application instance has 2 stream threads thus total 6 stream
threads across 3 instances of streams application cater to 6 partitions of
inputs topics.

Re: can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

["Matthias J. Sax" <mj...@apache.org>]

In general there is no ordering guarantee between topics. So it might 
depend a lot ofnthe details of your use case.

For example, if you know that it will be always two event, you could 
buffer the first one in a state-store, and wait for the second one to 
arrive and decide in which order to forward both events downstream for 
actual processing.


HTH, Matthias


On 3/20/23 11:57 PM, Pushkar Deole wrote:
> Hi All,
> 
> We have a kafka streams application that consumes from 2 different topics
> say topic A and topic B. The application uses data of telephone call on
> those topics and each call has a call id which is used as key to send
> events to those 2 topics. e.g. for a telephone call, the 1st event related
> to that call is sent to A with call id however subsequent event for that
> same call might go to topic B again with call id as key.
> 
> *At times, we need to process those 2 events in an order, which is not
> possible with the current topology that we are using*. *Can someone suggest
> if this is possible to achieve with streams?*
> The topology is as below:
> 
> Topic A has 6 partitions
> Topics B has 6 partitions
> Call id used as key on both topics
> Kafka streams application has 3 instances that consumes from both of the
> topics as source topics.
> Each streams application instance has 2 stream threads thus total 6 stream
> threads across 3 instances of streams application cater to 6 partitions of
> inputs topics.
>