You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sining Ma <si...@aol.com> on 2013/05/24 20:29:08 UTC
OffsetOutOfRangeException and getOffsetsBefore
Hi,
We are currently using kafka-0.7.1 right now.
I have two questions:
1. We use SimpleConsumer to aggregate messages to log files and there is no zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
And this exception happens when we start our consumer program. We do not know the reason why this happens.
How can I get a valid latest message offset in kafka-0.7.1 when this exception happens?
2. Before we start consumer, we call getOffsetsBefore function to get a list of valid offsets (up to maxSize) before the given time.
How can we interpret this list?
For example, this function returns an array [offset1, offset2].
Does this mean from offset1 to offset2 are valid, and offset2 to current offset are valid? We are confused about the meaning of this array.
-- Regards
Sining Ma
Re: OffsetOutOfRangeException and getOffsetsBefore
Posted by Suyog Rao <su...@loggly.com>.
If you use the Java API you can do SimpleConsumer.getOffsetsBefore(topic, partition, time, 1) which will return a long offset value
The time parameter can be kafka.api.OffsetRequest.EarliestTime() or kafka.api.OffsetRequest.LatestTime() based on your application need.
On May 24, 2013, at 2:03 PM, Sining Ma <si...@aol.com> wrote:
> Thanks Suyog
> Could you explain more about OffsetRequest?
> I find new
> OffsetRequest(topic: String, partition: Int, time: Long, maxNumOffsets: Int)
> in Kafka api.
>
>
> How can I send this request? And where can I receive a response from this OffsetRequest?
> Could you give me an example for this API?
>
>
>
> -- Regards
> Sining Ma
>
>
>
>
>
> -----Original Message-----
> From: Suyog Rao <su...@loggly.com>
> To: users <us...@kafka.apache.org>
> Sent: Fri, May 24, 2013 1:32 pm
> Subject: Re: OffsetOutOfRangeException and getOffsetsBefore
>
>
> Since you are using the simple consumer you will need to handle the
> OffsetOutOfRange Exception in your code. This happens when your consumer queries
> for an offset which is no longer persisted in Kafka (The logs have been deleted
> based on the retention policy). Ideally when this happens, the consumer should
> issue an OffsetRequest to get the latest/earliest offset currently available on
> the server. You can then use this in your FetchRequest. Whether you use the
> latest or earliest depends on your application needs. In essence, you are
> resetting the consumer to catch up with the server.
>
> On May 24, 2013, at 11:29 AM, Sining Ma <si...@aol.com> wrote:
>
>> Hi,
>> We are currently using kafka-0.7.1 right now.
>> I have two questions:
>> 1. We use SimpleConsumer to aggregate messages to log files and there is no
> zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
>> And this exception happens when we start our consumer program. We do not know
> the reason why this happens.
>> How can I get a valid latest message offset in kafka-0.7.1 when this exception
> happens?
>> 2. Before we start consumer, we call getOffsetsBefore function to get a list
> of valid offsets (up to maxSize) before the given time.
>> How can we interpret this list?
>> For example, this function returns an array [offset1, offset2].
>> Does this mean from offset1 to offset2 are valid, and offset2 to current
> offset are valid? We are confused about the meaning of this array.
>>
>>
>>
>> -- Regards
>> Sining Ma
>>
>>
>
>
>
Re: OffsetOutOfRangeException and getOffsetsBefore
Posted by Sining Ma <si...@aol.com>.
Thanks Suyog
Could you explain more about OffsetRequest?
I find new
OffsetRequest(topic: String, partition: Int, time: Long, maxNumOffsets: Int)
in Kafka api.
How can I send this request? And where can I receive a response from this OffsetRequest?
Could you give me an example for this API?
-- Regards
Sining Ma
-----Original Message-----
From: Suyog Rao <su...@loggly.com>
To: users <us...@kafka.apache.org>
Sent: Fri, May 24, 2013 1:32 pm
Subject: Re: OffsetOutOfRangeException and getOffsetsBefore
Since you are using the simple consumer you will need to handle the
OffsetOutOfRange Exception in your code. This happens when your consumer queries
for an offset which is no longer persisted in Kafka (The logs have been deleted
based on the retention policy). Ideally when this happens, the consumer should
issue an OffsetRequest to get the latest/earliest offset currently available on
the server. You can then use this in your FetchRequest. Whether you use the
latest or earliest depends on your application needs. In essence, you are
resetting the consumer to catch up with the server.
On May 24, 2013, at 11:29 AM, Sining Ma <si...@aol.com> wrote:
> Hi,
> We are currently using kafka-0.7.1 right now.
> I have two questions:
> 1. We use SimpleConsumer to aggregate messages to log files and there is no
zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
> And this exception happens when we start our consumer program. We do not know
the reason why this happens.
> How can I get a valid latest message offset in kafka-0.7.1 when this exception
happens?
> 2. Before we start consumer, we call getOffsetsBefore function to get a list
of valid offsets (up to maxSize) before the given time.
> How can we interpret this list?
> For example, this function returns an array [offset1, offset2].
> Does this mean from offset1 to offset2 are valid, and offset2 to current
offset are valid? We are confused about the meaning of this array.
>
>
>
> -- Regards
> Sining Ma
>
>
Re: OffsetOutOfRangeException and getOffsetsBefore
Posted by Suyog Rao <su...@loggly.com>.
Since you are using the simple consumer you will need to handle the OffsetOutOfRange Exception in your code. This happens when your consumer queries for an offset which is no longer persisted in Kafka (The logs have been deleted based on the retention policy). Ideally when this happens, the consumer should issue an OffsetRequest to get the latest/earliest offset currently available on the server. You can then use this in your FetchRequest. Whether you use the latest or earliest depends on your application needs. In essence, you are resetting the consumer to catch up with the server.
On May 24, 2013, at 11:29 AM, Sining Ma <si...@aol.com> wrote:
> Hi,
> We are currently using kafka-0.7.1 right now.
> I have two questions:
> 1. We use SimpleConsumer to aggregate messages to log files and there is no zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
> And this exception happens when we start our consumer program. We do not know the reason why this happens.
> How can I get a valid latest message offset in kafka-0.7.1 when this exception happens?
> 2. Before we start consumer, we call getOffsetsBefore function to get a list of valid offsets (up to maxSize) before the given time.
> How can we interpret this list?
> For example, this function returns an array [offset1, offset2].
> Does this mean from offset1 to offset2 are valid, and offset2 to current offset are valid? We are confused about the meaning of this array.
>
>
>
> -- Regards
> Sining Ma
>
>
Re: OffsetOutOfRangeException and getOffsetsBefore
Posted by Jun Rao <ju...@gmail.com>.
This gives you back two valid offsets, not a range.
Thanks,
Jun
On Fri, May 24, 2013 at 11:29 AM, Sining Ma <si...@aol.com> wrote:
> Hi,
> We are currently using kafka-0.7.1 right now.
> I have two questions:
> 1. We use SimpleConsumer to aggregate messages to log files and there is
> no zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException.
> And this exception happens when we start our consumer program. We do not
> know the reason why this happens.
> How can I get a valid latest message offset in kafka-0.7.1 when this
> exception happens?
> 2. Before we start consumer, we call getOffsetsBefore function to get a
> list of valid offsets (up to maxSize) before the given time.
> How can we interpret this list?
> For example, this function returns an array [offset1, offset2].
> Does this mean from offset1 to offset2 are valid, and offset2 to current
> offset are valid? We are confused about the meaning of this array.
>
>
>
> -- Regards
> Sining Ma
>
>
>