Kafka and Rx

[Marta Kacperek <M....@student.tudelft.nl>]

Dear Kafka developers and users,

I am currently working on creating a small library that would enable using Reactive Extensions (Rx) with Kafka as data source. I started with what seemed most reasonable for me: using Kafka's Java API and wrapping Kafka producers and consumers with RxJava interfaces (https://github.com/Netflix/RxJava/wiki).

One of my problems is version of your consumer API to use... Till now I have written basic bindings using version 0.8.1.1 and "high level consumer API", but then control over committing offsets is limited. On the other hand, I'm not so keen to invest my time in designing and writing library that would use this "simple/low level" API, dealing with partitioning, offsets and leader election, while it is soon to be replaced. 

For that reasons I am now attempting to use your new, not released yet, 0.9 consumer API - I have an impression that it is more or less stable at the moment, am I right? Can you estimate when will you finish and release version 0.9?

One on the suggestions I found in this document: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design is that new consumer client should enable non-blocking, asynchronous data access (and from what I've seen - it's not implemented yet). This is exactly what Reactive Extensions are about. Rx wrappers/bindings for Kafka API would give users very clean and convenient access to Kafka's data asynchronously, using observables (push-based collections). All kinds of operations on streams (modifications, filtering, merging and so on) are also enabled by Rx (see https://github.com/Netflix/RxJava/wiki for details).

Maybe you can consider adding Rx bindings as a new feature for Kafka consumer/producer API? Of course, that would require adding RxJava as dependency to Kafka client library, but RxJava is lightweight and has no dependencies of its own. And this would solve your design problem with non-blocking kafka access.

My library code is still under development, especially this using new consumer API (I have some troubles with testing, but this is a different thing...), and I would really appreciate some feedback from you at this stage. Any advice or comments and are welcome.

Best regards,
Marta Kacperek

Re: Kafka and Rx

[Neha Narkhede <ne...@gmail.com>]

For that reasons I am now attempting to use your new, not released yet, 0.9
consumer API - I have an impression that it is more or less stable at the
moment, am I right? Can you estimate when will you finish and release
version 0.9?

The API is checked into trunk and I would not rule out minor changes to it
before the 0.9 release. I suspect the 0.9 release is roughly 3 months away.

Thanks,
Neha


On Sun, Jun 15, 2014 at 11:28 PM, Marta Kacperek <
M.J.Kacperek@student.tudelft.nl> wrote:

> Dear Kafka developers and users,
>
> I am currently working on creating a small library that would enable using
> Reactive Extensions (Rx) with Kafka as data source. I started with what
> seemed most reasonable for me: using Kafka's Java API and wrapping Kafka
> producers and consumers with RxJava interfaces (
> https://github.com/Netflix/RxJava/wiki).
>
> One of my problems is version of your consumer API to use... Till now I
> have written basic bindings using version 0.8.1.1 and "high level consumer
> API", but then control over committing offsets is limited. On the other
> hand, I'm not so keen to invest my time in designing and writing library
> that would use this "simple/low level" API, dealing with partitioning,
> offsets and leader election, while it is soon to be replaced.
>
> For that reasons I am now attempting to use your new, not released yet,
> 0.9 consumer API - I have an impression that it is more or less stable at
> the moment, am I right? Can you estimate when will you finish and release
> version 0.9?
>
> One on the suggestions I found in this document:
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> is that new consumer client should enable non-blocking, asynchronous data
> access (and from what I've seen - it's not implemented yet). This is
> exactly what Reactive Extensions are about. Rx wrappers/bindings for Kafka
> API would give users very clean and convenient access to Kafka's data
> asynchronously, using observables (push-based collections). All kinds of
> operations on streams (modifications, filtering, merging and so on) are
> also enabled by Rx (see https://github.com/Netflix/RxJava/wiki for
> details).
>
> Maybe you can consider adding Rx bindings as a new feature for Kafka
> consumer/producer API? Of course, that would require adding RxJava as
> dependency to Kafka client library, but RxJava is lightweight and has no
> dependencies of its own. And this would solve your design problem with
> non-blocking kafka access.
>
> My library code is still under development, especially this using new
> consumer API (I have some troubles with testing, but this is a different
> thing...), and I would really appreciate some feedback from you at this
> stage. Any advice or comments and are welcome.
>
> Best regards,
> Marta Kacperek