You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Wang LongTian <lo...@outlook.com> on 2017/03/06 23:33:18 UTC
Can I create user defined stream processing function/api?
Dear folks,
Background: I'm leaning Kafka stream and want to use that in my product for real time streaming process with data from various sensors.
Question:
1. Can I define my own processing function/api in Kafka stream except the predefined functions like groupby(), count() etc.?
2. If I could define my own function, could you please specify a example or introduce me a github project for example?
Thank u in advance!
Regards,
Long Tian
Re: Can I create user defined stream processing function/api?
Posted by Michael Noll <mi...@confluent.io>.
There's also an end-to-end example for DSL and Processor API integration:
https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/test/java/io/confluent/examples/streams/MixAndMatchLambdaIntegrationTest.java
Best,
Michael
On Tue, Mar 7, 2017 at 4:51 PM, LongTian Wang <lo...@outlook.com> wrote:
> Really appreciated, Matthias!
> That's what I wanted.
>
> Regards,
> Long Tian
> ________________________________
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: 07 March 2017 07:48:08
> To: users@kafka.apache.org
> Subject: Re: Can I create user defined stream processing function/api?
>
> Hi,
>
> you can implements custom operator via process(), transform(), and
> transform() values.
>
> Also, if you want to have even more control over the topology, you can
> use low-level Processor API directly instead of DSL.
>
> http://docs.confluent.io/current/streams/developer-
> guide.html#processor-api
>
>
> -Matthias
>
> On 3/6/17 3:33 PM, Wang LongTian wrote:
> > Dear folks,
> >
> > Background: I'm leaning Kafka stream and want to use that in my product
> for real time streaming process with data from various sensors.
> >
> > Question:
> > 1. Can I define my own processing function/api in Kafka stream except
> the predefined functions like groupby(), count() etc.?
> > 2. If I could define my own function, could you please specify a example
> or introduce me a github project for example?
> >
> > Thank u in advance!
> >
> > Regards,
> > Long Tian
> >
>
>
Re: Can I create user defined stream processing function/api?
Posted by LongTian Wang <lo...@outlook.com>.
Really appreciated, Matthias!
That's what I wanted.
Regards,
Long Tian
________________________________
From: Matthias J. Sax <ma...@confluent.io>
Sent: 07 March 2017 07:48:08
To: users@kafka.apache.org
Subject: Re: Can I create user defined stream processing function/api?
Hi,
you can implements custom operator via process(), transform(), and
transform() values.
Also, if you want to have even more control over the topology, you can
use low-level Processor API directly instead of DSL.
http://docs.confluent.io/current/streams/developer-guide.html#processor-api
-Matthias
On 3/6/17 3:33 PM, Wang LongTian wrote:
> Dear folks,
>
> Background: I'm leaning Kafka stream and want to use that in my product for real time streaming process with data from various sensors.
>
> Question:
> 1. Can I define my own processing function/api in Kafka stream except the predefined functions like groupby(), count() etc.?
> 2. If I could define my own function, could you please specify a example or introduce me a github project for example?
>
> Thank u in advance!
>
> Regards,
> Long Tian
>
Re: Can I create user defined stream processing function/api?
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi,
you can implements custom operator via process(), transform(), and
transform() values.
Also, if you want to have even more control over the topology, you can
use low-level Processor API directly instead of DSL.
http://docs.confluent.io/current/streams/developer-guide.html#processor-api
-Matthias
On 3/6/17 3:33 PM, Wang LongTian wrote:
> Dear folks,
>
> Background: I'm leaning Kafka stream and want to use that in my product for real time streaming process with data from various sensors.
>
> Question:
> 1. Can I define my own processing function/api in Kafka stream except the predefined functions like groupby(), count() etc.?
> 2. If I could define my own function, could you please specify a example or introduce me a github project for example?
>
> Thank u in advance!
>
> Regards,
> Long Tian
>