You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by pradeep s <sr...@gmail.com> on 2022/01/12 04:46:01 UTC
Kafka streams usecase
Hi ,
I have a requirement to stream item details to specific destinations .
There are three different kafka streams , one for item info, second for
item price and promotions and third for item availability .
I want to join all these info and produce a single message containing
item,price and availability .
Can kafka streams be leveraged for this , if all the messages across three
topics can be joined using a coming item identifier . Also its not
necessary that all three topics have data .
For example, if item setup is done and no inventory is allocated, only item
and price topics will have data . Also any good pointers to see a sample
app for joining multiple streams and producing a single message to a new
topic.
Thanks
Pradeep
Re: Kafka streams usecase
Posted by Chad Preisler <ch...@gmail.com>.
It can be done with the consumer API. However, you're just going to end up
re-implementing what is already there in the streams DSL. It will be far
easier to use the Stream DSL join functionality to accomplish this. I've
never tried to do it with a simple consumer.
On Wed, Feb 16, 2022 at 6:45 PM pradeep s <sr...@gmail.com>
wrote:
> Thanks Chad! if we want to consume from multiple topics and persist to a
> database , can i go with a consumer and lookup the record and update
> .Requirement is to consume from item topic and price topic and create a
> record in postgress . Both topic have item id in message which is the key
> in postgress database . Can this be done with a simple consumer ?
>
> On Thu, Jan 13, 2022 at 11:11 AM Chad Preisler <ch...@gmail.com>
> wrote:
>
> > Yes Kafka streams can be used to do this. There are probably several ways
> > to implement this. We did something like this in Java using a
> groupByKey()
> > and reduce() functions. The three topics we wanted to combine into one
> > topic had different schemas and different java class types. So to combine
> > them together into one aggregated object we did the following.
> >
> > - Create a class with data members of the three objects we wanted to
> > combine. Let's call it AggregateObj.
> > - Create a KStream for each topic we wanted to combine.
> > - For each KStream use a map function that creates and outputs an
> > AggregateObj setting the input stream object to the correct data member
> on
> > the AggregateObj.
> > - Create an intermediate topic to write individual AggregateObj from each
> > KStream.
> > - Create a stream to read the intermediate topic and use the groupByKey()
> > and reduce() function to create one AggregateObj that has all the parts.
> > Output that result to the final combined output stream using
> > toStream().to().
> >
> > We did all of this in one application. You may be able to accomplish the
> > same thing using aggregate a different way or you may be able to use left
> > join methods to accomplish the same thing. I can't share the code. Sorry.
> >
> > On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sr...@gmail.com>
> > wrote:
> >
> > > Hi ,
> > > I have a requirement to stream item details to specific destinations .
> > > There are three different kafka streams , one for item info, second for
> > > item price and promotions and third for item availability .
> > > I want to join all these info and produce a single message containing
> > > item,price and availability .
> > > Can kafka streams be leveraged for this , if all the messages across
> > three
> > > topics can be joined using a coming item identifier . Also its not
> > > necessary that all three topics have data .
> > > For example, if item setup is done and no inventory is allocated, only
> > item
> > > and price topics will have data . Also any good pointers to see a
> sample
> > > app for joining multiple streams and producing a single message to a
> new
> > > topic.
> > > Thanks
> > > Pradeep
> > >
> >
>
Re: Kafka streams usecase
Posted by pradeep s <sr...@gmail.com>.
Thanks Chad! if we want to consume from multiple topics and persist to a
database , can i go with a consumer and lookup the record and update
.Requirement is to consume from item topic and price topic and create a
record in postgress . Both topic have item id in message which is the key
in postgress database . Can this be done with a simple consumer ?
On Thu, Jan 13, 2022 at 11:11 AM Chad Preisler <ch...@gmail.com>
wrote:
> Yes Kafka streams can be used to do this. There are probably several ways
> to implement this. We did something like this in Java using a groupByKey()
> and reduce() functions. The three topics we wanted to combine into one
> topic had different schemas and different java class types. So to combine
> them together into one aggregated object we did the following.
>
> - Create a class with data members of the three objects we wanted to
> combine. Let's call it AggregateObj.
> - Create a KStream for each topic we wanted to combine.
> - For each KStream use a map function that creates and outputs an
> AggregateObj setting the input stream object to the correct data member on
> the AggregateObj.
> - Create an intermediate topic to write individual AggregateObj from each
> KStream.
> - Create a stream to read the intermediate topic and use the groupByKey()
> and reduce() function to create one AggregateObj that has all the parts.
> Output that result to the final combined output stream using
> toStream().to().
>
> We did all of this in one application. You may be able to accomplish the
> same thing using aggregate a different way or you may be able to use left
> join methods to accomplish the same thing. I can't share the code. Sorry.
>
> On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sr...@gmail.com>
> wrote:
>
> > Hi ,
> > I have a requirement to stream item details to specific destinations .
> > There are three different kafka streams , one for item info, second for
> > item price and promotions and third for item availability .
> > I want to join all these info and produce a single message containing
> > item,price and availability .
> > Can kafka streams be leveraged for this , if all the messages across
> three
> > topics can be joined using a coming item identifier . Also its not
> > necessary that all three topics have data .
> > For example, if item setup is done and no inventory is allocated, only
> item
> > and price topics will have data . Also any good pointers to see a sample
> > app for joining multiple streams and producing a single message to a new
> > topic.
> > Thanks
> > Pradeep
> >
>
Re: Kafka streams usecase
Posted by Chad Preisler <ch...@gmail.com>.
Yes Kafka streams can be used to do this. There are probably several ways
to implement this. We did something like this in Java using a groupByKey()
and reduce() functions. The three topics we wanted to combine into one
topic had different schemas and different java class types. So to combine
them together into one aggregated object we did the following.
- Create a class with data members of the three objects we wanted to
combine. Let's call it AggregateObj.
- Create a KStream for each topic we wanted to combine.
- For each KStream use a map function that creates and outputs an
AggregateObj setting the input stream object to the correct data member on
the AggregateObj.
- Create an intermediate topic to write individual AggregateObj from each
KStream.
- Create a stream to read the intermediate topic and use the groupByKey()
and reduce() function to create one AggregateObj that has all the parts.
Output that result to the final combined output stream using
toStream().to().
We did all of this in one application. You may be able to accomplish the
same thing using aggregate a different way or you may be able to use left
join methods to accomplish the same thing. I can't share the code. Sorry.
On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sr...@gmail.com>
wrote:
> Hi ,
> I have a requirement to stream item details to specific destinations .
> There are three different kafka streams , one for item info, second for
> item price and promotions and third for item availability .
> I want to join all these info and produce a single message containing
> item,price and availability .
> Can kafka streams be leveraged for this , if all the messages across three
> topics can be joined using a coming item identifier . Also its not
> necessary that all three topics have data .
> For example, if item setup is done and no inventory is allocated, only item
> and price topics will have data . Also any good pointers to see a sample
> app for joining multiple streams and producing a single message to a new
> topic.
> Thanks
> Pradeep
>