Kafka offset out of range using SimpleConsumer Example

[Ryan Thompson <ry...@uptake.com>]

Hello,

I've implemented something quite similar to the SimpleConsumer example on
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

I'm using it to traverse a specific range of offsets.

I find that sometimes, in the middle of this traversal, I end up hitting an
"Offset out of range" error (error code 1).  The offset that's being
requested when this happens is determined by a previous message's
nextOffset(), as in the example.

When this happens (and it doesn't always happen), I find that this offset
is not even close to being in the range I am attempting to traverse.  For
instance, in one case, I was traversing the range (1665615779, 1861334452),
but ended up requesting offset 1193311 (again, based on a message's
nextOffset() value, in the middle of traversal).

Two questions:
1- What could explain this behavior (message.nextOffset() being out of the
range I am traversing)
2- Is there any more robust way to recover from this other than simply
bailing, as in the example?

Thanks,
Ryan

Re: Kafka offset out of range using SimpleConsumer Example

[Ryan Thompson <ry...@uptake.com>]

#1 turned out to be invalid, my logging was simply bad (I was logging the
number of messages I'd read so far for that partition, not the requested
read offset)

#2 is still valid, though.  I'm thinking that a possible explanation might
be that the part of the log I was processing was deleted, but I'm not sure
yet.

Thanks,
Ryan

On Thu, Aug 11, 2016 at 9:23 PM, Ryan Thompson <ry...@uptake.com>
wrote:

> Hello,
>
> I've implemented something quite similar to the SimpleConsumer example on
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.
> 0+SimpleConsumer+Example
>
> I'm using it to traverse a specific range of offsets.
>
> I find that sometimes, in the middle of this traversal, I end up hitting
> an "Offset out of range" error (error code 1).  The offset that's being
> requested when this happens is determined by a previous message's
> nextOffset(), as in the example.
>
> When this happens (and it doesn't always happen), I find that this offset
> is not even close to being in the range I am attempting to traverse.  For
> instance, in one case, I was traversing the range (1665615779, 1861334452),
> but ended up requesting offset 1193311 (again, based on a message's
> nextOffset() value, in the middle of traversal).
>
> Two questions:
> 1- What could explain this behavior (message.nextOffset() being out of the
> range I am traversing)
> 2- Is there any more robust way to recover from this other than simply
> bailing, as in the example?
>
> Thanks,
> Ryan
>