You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by David Arthur <mu...@gmail.com> on 2013/02/21 03:31:41 UTC
Initializing fetch offsets
Is there currently a way to get the latest offset (or close to the
latest) for a partition? Is this covered by the Offset API, and if so
could someone explain this API a bit?
I'd like to write a consumer that can start from 0 (like unix cat) or
start from the most recent (like unix tail).
Thanks!
-David
Re: Initializing fetch offsets
Posted by Neha Narkhede <ne...@gmail.com>.
Thank you for the feedback on documentation and also for improving it. That
is the best way to have effective documentation.
Thanks,
Neha
On Thu, Feb 21, 2013 at 8:53 AM, Colin Blower <cb...@barracuda.com> wrote:
> In the old protocol description the -1 and -2 values are explicitly
> mentioned.
> https://cwiki.apache.org/**confluence/display/KAFKA/Wire+**Format<https://cwiki.apache.org/confluence/display/KAFKA/Wire+Format>
>
> It looks like it had been left out of the new 0.8 docs, however I just
> added it.
> https://cwiki.apache.org/**confluence/display/KAFKA/A+**
> Guide+To+The+Kafka+Protocol#**AGuideToTheKafkaProtocol-**OffsetAPI<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI>
>
> Hopefully the docs are more helpful now. Feel free to ask for further
> clarifications.
>
>
>
> On 02/21/2013 05:04 AM, David Arthur wrote:
>
>> Yes, I think that section of the protocol guide could be improved.
>> Specifically, it would be useful to explain what the " Time" and
>> "MaxNumberOfOffsets" parameters are used for, as well documenting these
>> two magic numbers (-1 and -2).
>>
>> -David
>>
>> On 2/21/13 12:13 AM, Neha Narkhede wrote:
>>
>>> David,
>>>
>>> The API that you are looking for is OffsetRequest. If you pass in -2 as
>>> the
>>> value for Time, it will get you the earliest offset available for that
>>> partition. If you pass in -1, it will get you the latest offset.
>>>
>>> You can find more information about the OffsetRequest and OffsetResponse
>>> APIs here -
>>> https://cwiki.apache.org/**confluence/display/KAFKA/A+**
>>> Guide+To+The+Kafka+Protocol#**AGuideToTheKafkaProtocol-**OffsetAPI<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI>
>>>
>>> Please let us know if that description can be improved.
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <mu...@gmail.com> wrote:
>>>
>>> Is there currently a way to get the latest offset (or close to the
>>>> latest)
>>>> for a partition? Is this covered by the Offset API, and if so could
>>>> someone
>>>> explain this API a bit?
>>>>
>>>> I'd like to write a consumer that can start from 0 (like unix cat) or
>>>> start from the most recent (like unix tail).
>>>>
>>>> Thanks!
>>>> -David
>>>>
>>>>
>
> --
> *Colin Blower*
> /Software Engineer/
> Barracuda Networks Inc.
> +1 408-342-5576 (o)
>
Re: Initializing fetch offsets
Posted by Colin Blower <cb...@barracuda.com>.
In the old protocol description the -1 and -2 values are explicitly
mentioned.
https://cwiki.apache.org/confluence/display/KAFKA/Wire+Format
It looks like it had been left out of the new 0.8 docs, however I just
added it.
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
Hopefully the docs are more helpful now. Feel free to ask for further
clarifications.
On 02/21/2013 05:04 AM, David Arthur wrote:
> Yes, I think that section of the protocol guide could be improved.
> Specifically, it would be useful to explain what the " Time" and
> "MaxNumberOfOffsets" parameters are used for, as well documenting these
> two magic numbers (-1 and -2).
>
> -David
>
> On 2/21/13 12:13 AM, Neha Narkhede wrote:
>> David,
>>
>> The API that you are looking for is OffsetRequest. If you pass in -2 as the
>> value for Time, it will get you the earliest offset available for that
>> partition. If you pass in -1, it will get you the latest offset.
>>
>> You can find more information about the OffsetRequest and OffsetResponse
>> APIs here -
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
>>
>> Please let us know if that description can be improved.
>>
>> Thanks,
>> Neha
>>
>>
>> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <mu...@gmail.com> wrote:
>>
>>> Is there currently a way to get the latest offset (or close to the latest)
>>> for a partition? Is this covered by the Offset API, and if so could someone
>>> explain this API a bit?
>>>
>>> I'd like to write a consumer that can start from 0 (like unix cat) or
>>> start from the most recent (like unix tail).
>>>
>>> Thanks!
>>> -David
>>>
--
*Colin Blower*
/Software Engineer/
Barracuda Networks Inc.
+1 408-342-5576 (o)
Re: Initializing fetch offsets
Posted by David Arthur <mu...@gmail.com>.
Yes, I think that section of the protocol guide could be improved.
Specifically, it would be useful to explain what the " Time" and
"MaxNumberOfOffsets" parameters are used for, as well documenting these
two magic numbers (-1 and -2).
-David
On 2/21/13 12:13 AM, Neha Narkhede wrote:
> David,
>
> The API that you are looking for is OffsetRequest. If you pass in -2 as the
> value for Time, it will get you the earliest offset available for that
> partition. If you pass in -1, it will get you the latest offset.
>
> You can find more information about the OffsetRequest and OffsetResponse
> APIs here -
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
>
> Please let us know if that description can be improved.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <mu...@gmail.com> wrote:
>
>> Is there currently a way to get the latest offset (or close to the latest)
>> for a partition? Is this covered by the Offset API, and if so could someone
>> explain this API a bit?
>>
>> I'd like to write a consumer that can start from 0 (like unix cat) or
>> start from the most recent (like unix tail).
>>
>> Thanks!
>> -David
>>
Re: Initializing fetch offsets
Posted by Neha Narkhede <ne...@gmail.com>.
David,
The API that you are looking for is OffsetRequest. If you pass in -2 as the
value for Time, it will get you the earliest offset available for that
partition. If you pass in -1, it will get you the latest offset.
You can find more information about the OffsetRequest and OffsetResponse
APIs here -
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
Please let us know if that description can be improved.
Thanks,
Neha
On Wed, Feb 20, 2013 at 6:31 PM, David Arthur <mu...@gmail.com> wrote:
> Is there currently a way to get the latest offset (or close to the latest)
> for a partition? Is this covered by the Offset API, and if so could someone
> explain this API a bit?
>
> I'd like to write a consumer that can start from 0 (like unix cat) or
> start from the most recent (like unix tail).
>
> Thanks!
> -David
>