Re: running kafka streams inside kafka connect

[Guozhang Wang <wa...@gmail.com>]

Hi RK,

You may find this blog post helpful for incorporating Streams with Connect:

https://yokota.blog/2018/04/23/embedding-kafka-connect-in-kafka-streams-ksql/

I've also cc'ed Robert, the author of this blog post if you are interested
in his code and want to learn more.


Guozhang

On Tue, Oct 30, 2018 at 8:54 PM RK Sing <dh...@gmail.com> wrote:

> Thank you that's helpful.
>
> So that means  resource management of streams needs to a separate concern
> than that of Kafka connect
>
> On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Streams is not designed to be run inside Connect, and this won't work.
> >
> > What you can do is, to import the data via connect into a "staging
> > topic" and then read this "staging topic" with a Kafka Streams
> > application and apply the transformations etc to write the data into the
> > actual target topics.
> >
> >
> > -Matthias
> >
> > On 10/25/18 2:34 PM, RK Sing wrote:
> > > Thank you Ryanne for the answer, My question is can I run Streams app
> > > inside Connect, what I mean is since Streams is a library, can I build
> a
> > > custom Source connector and use Streams in there . or using Streams in
> > > transforms??
> > > We want to containerize Kafka connect as in wrap it in a docker, and
> > since
> > > all the Kafka-connect configs are stored in Kafka itself, so the
> > container
> > > can be stateless, we can spin up containers as and when needed, we want
> > > this approach to fetch data from microservices and golden gate oracle.
> > etc.
> > >
> > >
> > >
> > >
> > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan <ry...@gmail.com>
> > wrote:
> > >
> > >> Dhurandar, definitely! Connect and Streams are both agnostic to how
> > their
> > >> workers are run. They aren't really platforms per se. You just need to
> > spin
> > >> up one or more workers and they do their thing. So a Streams app
> doesn't
> > >> run "inside" Connect, but you can certainly have Connect and Streams
> > >> workers talk to each other through Kafka topics.
> > >>
> > >> Ryanne
> > >>
> > >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing <dh...@gmail.com>
> > wrote:
> > >>
> > >>> We have a requirement to do Single row transformations, basic joins,
> > >>> deduping and routing from the source to the destination Kafka topics.
> > >>>
> > >>> We want to use Kafka-connect as the platform which is running Kafka
> > >> stream
> > >>> inside. Has anyone used Kafkastreams inside Kafka connect ??
> > >>>
> > >>> Is this pattern ok for Kafka-connect?
> > >>>
> > >>> regards
> > >>> dhurandar
> > >>>
> > >>
> > >
> >
> >
>


-- 
-- Guozhang