Java 8 influence on next-generation Java producer and consumer APIs

[Clark Breyman <cl...@breyman.com>]

Jay et al,

What are your current thoughts on ensuring that the next-generation APIs
play nicely with both lambdas and the extensions to the standard runtime in
Java 8?

My thoughts are that if folks are doing the work to reimplement/redesign
the API, it should be as compatible as possible with the latest stuff
provided it doesn't break 1.6 compatibility (as long as that's required).

Areas where this might influence the design:
- Using single-method interfaces that could be supplied by lambdas when
running in J8
- Consideration of Spliterator, Supplier, Stream, ... when designing their
Kakfa counterparts.

Regards,
Clark

Re: Java 8 influence on next-generation Java producer and consumer APIs

[Jay Kreps <ja...@gmail.com>]

I think this is a good point and you are not the first person to bring it
up.

I am not hugely knowledgable about java 8 so any feedback would be useful.

In the producer I think the biggest impact is that the Callback can be
implemented with a lambda instead of a anon class which will be much nicer.
But maybe there are other things? Would be great if someone could take a
look at that producer api with a forward looking eye...

In the consumer there is potentially a lot of similarity between Stream and
the consumer. What we had tentatively proposed in this area was actually to
replace the iterator we currently provide with a lower-level method
   List<RecordAndMetadata> poll(long timeoutMs)
This method polls over all topics and returns a list of messages for
processing. This is arguably the least convenient end-user api for some of
the simpler cases, but the advantage is that it can be implemented in a
completely single threaded fashion which allows us to full decouple the
threading model of message processing--currently the zookeeper consumer
actually requires a per-topic blocking thread. There are a number of
slightly more convenient apis that could be implemented on top of this. I
think we were just about to kick off a more detailed discussion here...

-Jay






On Wed, Jan 29, 2014 at 12:05 PM, Clark Breyman <cl...@breyman.com> wrote:

> Jay et al,
>
> What are your current thoughts on ensuring that the next-generation APIs
> play nicely with both lambdas and the extensions to the standard runtime in
> Java 8?
>
> My thoughts are that if folks are doing the work to reimplement/redesign
> the API, it should be as compatible as possible with the latest stuff
> provided it doesn't break 1.6 compatibility (as long as that's required).
>
> Areas where this might influence the design:
> - Using single-method interfaces that could be supplied by lambdas when
> running in J8
> - Consideration of Spliterator, Supplier, Stream, ... when designing their
> Kakfa counterparts.
>
> Regards,
> Clark
>