KStreams and multiple instance

[Pietro Galassi <pi...@gmail.com>]

Hi all,
hi have hope you can help me figure out this scenario.

I have a multiinstance microservice that consumes from a topic
(ordersTopic) all of them use the same consumer_group.

This microservice uses a KStream to aggregate (sum) topic events and
produces results on another topic (countTopic).

Have two questions:

1) Can i have problems on counts due to multiple instance of the same
microservies ?
2) I need rockDB and materialized view in order to store data ?

Thanks a lot.
Regards,
Pietro Galassi

Re: KStreams and multiple instance

[Guozhang Wang <wa...@gmail.com>]

Hello Pietro,

1) If you are using the Streams DSL with an aggregation, it would
repartition the input streams by the aggregation field for data
parallelism, and hence multiple instances would be able to do the
aggregation in parallel and independently with correct results.
2) Short answer is "probably": you can use different state store impls for
materializing the running aggregates.

You can read some general docs for this if you've further questions:
https://kafka.apache.org/28/documentation/streams/architecture


Guozhang

On Thu, May 6, 2021 at 8:40 AM Pietro Galassi <pi...@gmail.com>
wrote:

> Hi all,
> hi have hope you can help me figure out this scenario.
>
> I have a multiinstance microservice that consumes from a topic
> (ordersTopic) all of them use the same consumer_group.
>
> This microservice uses a KStream to aggregate (sum) topic events and
> produces results on another topic (countTopic).
>
> Have two questions:
>
> 1) Can i have problems on counts due to multiple instance of the same
> microservies ?
> 2) I need rockDB and materialized view in order to store data ?
>
> Thanks a lot.
> Regards,
> Pietro Galassi
>


-- 
-- Guozhang