Kafka Streams - exposing processor name in DSL

[Jon Bender <jo...@gmail.com>]

Hi!

I was trying to build a topology using the DSL, add a custom processor and
then add a sink for it.  However, it seems like there's no great way to do
this using the APIs, since the processor's internal "name" is not exposed
in the KStream interface, and the .process method doesn't return the latest
KStream.

Is it possible to expose the underlying processor name from the builder
APIs, so that we could connect custom processors by referencing their names
directly?  I suppose you could also just make the return type of .process
into a KStream as well.

Cheers,
Jon

Re: Kafka Streams - exposing processor name in DSL

[Jon Bender <jo...@gmail.com>]

Guozhang,

Thanks for the detailed response, makes sense!  Created
https://issues.apache.org/jira/browse/KAFKA-3455

Cheers,
Jon

On Wed, Mar 23, 2016 at 11:35 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hello Jon,
>
> Thanks for the feedback. This is definitely something we wanted to support
> in the Streams DSL.
>
> One tricky thing, though, is that some operations do not translate to a
> single processor, but a sub-graph of processors (think of a stream-stream
> join, which is translated to actually 5 processors for windowing / state
> queries / merging, each with a different internal name). So how to define
> the API to return the processor name needs some more thinking.
>
> That said, we should definitely try to add it soon. Could you file a
> sub-task under https://issues.apache.org/jira/browse/KAFKA-2590 to keep
> track of this feature?
>
> Guozhang
>
>
>
> On Wed, Mar 23, 2016 at 10:17 AM, Jon Bender <jo...@gmail.com>
> wrote:
>
> > Hi!
> >
> > I was trying to build a topology using the DSL, add a custom processor
> and
> > then add a sink for it.  However, it seems like there's no great way to
> do
> > this using the APIs, since the processor's internal "name" is not exposed
> > in the KStream interface, and the .process method doesn't return the
> latest
> > KStream.
> >
> > Is it possible to expose the underlying processor name from the builder
> > APIs, so that we could connect custom processors by referencing their
> names
> > directly?  I suppose you could also just make the return type of .process
> > into a KStream as well.
> >
> > Cheers,
> > Jon
> >
>
>
>
> --
> -- Guozhang
>

Re: Kafka Streams - exposing processor name in DSL

[Guozhang Wang <wa...@gmail.com>]

Hello Jon,

Thanks for the feedback. This is definitely something we wanted to support
in the Streams DSL.

One tricky thing, though, is that some operations do not translate to a
single processor, but a sub-graph of processors (think of a stream-stream
join, which is translated to actually 5 processors for windowing / state
queries / merging, each with a different internal name). So how to define
the API to return the processor name needs some more thinking.

That said, we should definitely try to add it soon. Could you file a
sub-task under https://issues.apache.org/jira/browse/KAFKA-2590 to keep
track of this feature?

Guozhang



On Wed, Mar 23, 2016 at 10:17 AM, Jon Bender <jo...@gmail.com>
wrote:

> Hi!
>
> I was trying to build a topology using the DSL, add a custom processor and
> then add a sink for it.  However, it seems like there's no great way to do
> this using the APIs, since the processor's internal "name" is not exposed
> in the KStream interface, and the .process method doesn't return the latest
> KStream.
>
> Is it possible to expose the underlying processor name from the builder
> APIs, so that we could connect custom processors by referencing their names
> directly?  I suppose you could also just make the return type of .process
> into a KStream as well.
>
> Cheers,
> Jon
>



-- 
-- Guozhang