You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Robert Quinlivan <rq...@signal.co> on 2017/05/17 20:45:39 UTC

Java 8 stream consumer pattern

Hello,

I've been looking at writing a Java 8 streams API wrapper for the Kafka
consumer. Since this seems like a common use case I was wondering if
someone in the user community had already begun a project like this.

My goal is to be able to get back a Stream<ConsumerRecord<K, V>> wrapping
the results of #poll() which can then be passed into a map/filter pipeline.
I am using an underlying blocking queue data structure to buffer in memory
and using Stream.generate() to pull records. Any recommendations on a best
approach here?

Thanks
-- 
Robert Quinlivan
Software Engineer, Signal

Re: Java 8 stream consumer pattern

Posted by Robert Quinlivan <rq...@signal.co>.

Thanks, Matthias, I had not considered this. I will take a look at the
streams use case.

Thanks

On Wed, May 17, 2017 at 4:58 PM, Matthias J. Sax <ma...@confluent.io>
wrote:

> Did you try out Kafka Streams API instead of wrapping the consumer? It
> does support Lambdas already:
>
> https://github.com/confluentinc/examples/blob/3.
> 2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
> MapFunctionLambdaExample.java#L126
>
> Full docs: http://docs.confluent.io/current/streams/index.html
>
>
> -Matthias
>
> On 5/17/17 1:45 PM, Robert Quinlivan wrote:
> > Hello,
> >
> > I've been looking at writing a Java 8 streams API wrapper for the Kafka
> > consumer. Since this seems like a common use case I was wondering if
> > someone in the user community had already begun a project like this.
> >
> > My goal is to be able to get back a Stream<ConsumerRecord<K, V>> wrapping
> > the results of #poll() which can then be passed into a map/filter
> pipeline.
> > I am using an underlying blocking queue data structure to buffer in
> memory
> > and using Stream.generate() to pull records. Any recommendations on a
> best
> > approach here?
> >
> > Thanks
> >
>
>


-- 
Robert Quinlivan
Software Engineer, Signal

Re: Java 8 stream consumer pattern

Posted by "Matthias J. Sax" <ma...@confluent.io>.

Did you try out Kafka Streams API instead of wrapping the consumer? It
does support Lambdas already:

https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/main/java/io/confluent/examples/streams/MapFunctionLambdaExample.java#L126

Full docs: http://docs.confluent.io/current/streams/index.html


-Matthias

On 5/17/17 1:45 PM, Robert Quinlivan wrote:
> Hello,
> 
> I've been looking at writing a Java 8 streams API wrapper for the Kafka
> consumer. Since this seems like a common use case I was wondering if
> someone in the user community had already begun a project like this.
> 
> My goal is to be able to get back a Stream<ConsumerRecord<K, V>> wrapping
> the results of #poll() which can then be passed into a map/filter pipeline.
> I am using an underlying blocking queue data structure to buffer in memory
> and using Stream.generate() to pull records. Any recommendations on a best
> approach here?
> 
> Thanks
>