Seek to invalid offset, new consumer

[Giidox <ak...@marmelandia.com>]

Hi all!

I am using the new consumer API by manually assigning partitions. I’m having some trouble with seek.

When I seek with a valid offset, poll works ok. However, if I call seek with an offset that is so small that the broker no longer has that offset, poll returns no records. Is there a way to get a callback or exception for seeking to an invalid offset? Is there a way to discover what the valid offset range is, with the new consumer?

Is the old consumer API deprecated (or is it planned to be deprecated)?

Any help is much appreciated.

- Giidox

Re: Seek to invalid offset, new consumer

[Guozhang Wang <wa...@gmail.com>]

Hi Giidox:

Actually I just realized that position() call will trigger the underlying
select() function when necessary, so your proposal above actually works.

Guozhang

On Mon, Mar 7, 2016 at 4:57 PM, Guozhang Wang <wa...@gmail.com> wrote:

> consumer.seekToBeginning / seekToEnd evaluates lazily, and hence you have
> to call poll in order to really update the fetching position. The first
> record's offset would then be the starting offset.
>
> Guozhang
>
>
> On Mon, Mar 7, 2016 at 4:53 AM, Giidox <ak...@marmelandia.com> wrote:
>
>> Excellent, many thanks!
>>
>> With “auto.offset.reset” as “none”, I get OffsetOutOfRangeException,
>> which is what I want.
>>
>> To find out how many messages the client I had missed, I called
>> consumer.seekToBeginning and consumer.position (without calling poll). Is
>> this a correct way to find out the first available offset?
>>
>> Best regards,
>>
>> Giidox
>>
>>
>>
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Re: Seek to invalid offset, new consumer

[Guozhang Wang <wa...@gmail.com>]

consumer.seekToBeginning / seekToEnd evaluates lazily, and hence you have
to call poll in order to really update the fetching position. The first
record's offset would then be the starting offset.

Guozhang


On Mon, Mar 7, 2016 at 4:53 AM, Giidox <ak...@marmelandia.com> wrote:

> Excellent, many thanks!
>
> With “auto.offset.reset” as “none”, I get OffsetOutOfRangeException, which
> is what I want.
>
> To find out how many messages the client I had missed, I called
> consumer.seekToBeginning and consumer.position (without calling poll). Is
> this a correct way to find out the first available offset?
>
> Best regards,
>
> Giidox
>
>
>


-- 
-- Guozhang

Re: Seek to invalid offset, new consumer

[Giidox <ak...@marmelandia.com>]

Excellent, many thanks!

With “auto.offset.reset” as “none”, I get OffsetOutOfRangeException, which is what I want.

To find out how many messages the client I had missed, I called consumer.seekToBeginning and consumer.position (without calling poll). Is this a correct way to find out the first available offset?

Best regards, 

Giidox

Re: Seek to invalid offset, new consumer

[Guozhang Wang <wa...@gmail.com>]

Hello,

1) When the fetch offset is out of range, broker will send back the
corresponding error code and the new consumer should reset its offset
automatically according to the reset policy (config "auto.offset.reset"),
by default it is set to "latest", i.e. then log end offset. If you do not
have any new messages coming in then consumer will not return anything.

Try set the config to "earliest" to start from log start offset and see if
there is any messages returned from poll().

2) We are planning to gradually deprecate the old consumer APIs, but it is
still under discussion which release we will mark it as deprecated and the
new consumer API as "stable".

Guozhang



On Wed, Mar 2, 2016 at 4:15 AM, Giidox <ak...@marmelandia.com> wrote:

> Hi all!
>
> I am using the new consumer API by manually assigning partitions. I’m
> having some trouble with seek.
>
> When I seek with a valid offset, poll works ok. However, if I call seek
> with an offset that is so small that the broker no longer has that offset,
> poll returns no records. Is there a way to get a callback or exception for
> seeking to an invalid offset? Is there a way to discover what the valid
> offset range is, with the new consumer?
>
> Is the old consumer API deprecated (or is it planned to be deprecated)?
>
> Any help is much appreciated.
>
> - Giidox
>
>


-- 
-- Guozhang