kafka consumer group API

[Shashank Singh <sh...@gmail.com>]

Hi Team

I was going over the documentation and links related to consumer API
provided in
0.8.2.1 release
I got to know of major improvement in consumer API in 0.8.3 release. May I
know when we are planning to have this release.

One more help - I was following up the example given on
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

In this approach we can't decide
a) how to get the current offset
b) how to set a particular consumer group to start reading from a specific
offset

Also how to use the same example to read one message at a time from a topic
with many topics. I tried with creating a list of iterators and then
checking for hasNext() but that call in blocking. Even though I am using a
timeout on that call this seems not be working.

Instead what I have settled with is an internal queue to which the
multi-threaded consumer offers the messages into. Poll API polls the
internal queue. Problem is in case client code gets down without consuming
all the messages the offset is already set and the messages which have
already been read from iterator and put in queue will be never seen again.


-- 

*Warm Regards,*

*Shashank  *

*Mobile: +91 9910478553 *

*Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/
<http://in.linkedin.com/pub/shashank-singh/13/763/906/>*

Re: kafka consumer group API

[noah <ia...@gmail.com>]

Hi!

I did something similar. You can use the high level consumer but turn off
auto commit and commit only what you are done with. Here's the code I used:
https://github.com/iamnoah/kakfa-offsets-test

On Thu, Jul 9, 2015 at 4:53 PM Shashank Singh <sh...@gmail.com>
wrote:

> Hi Team
>
> I was going over the documentation and links related to consumer API
> provided in
> 0.8.2.1 release
> I got to know of major improvement in consumer API in 0.8.3 release. May I
> know when we are planning to have this release.
>
> One more help - I was following up the example given on
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>
> In this approach we can't decide
> a) how to get the current offset
> b) how to set a particular consumer group to start reading from a specific
> offset
>
> Also how to use the same example to read one message at a time from a topic
> with many topics. I tried with creating a list of iterators and then
> checking for hasNext() but that call in blocking. Even though I am using a
> timeout on that call this seems not be working.
>
> Instead what I have settled with is an internal queue to which the
> multi-threaded consumer offers the messages into. Poll API polls the
> internal queue. Problem is in case client code gets down without consuming
> all the messages the offset is already set and the messages which have
> already been read from iterator and put in queue will be never seen again.
>
>
> --
>
> *Warm Regards,*
>
> *Shashank  *
>
> *Mobile: +91 9910478553 *
>
> *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/
> <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*
>