You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark <st...@gmail.com> on 2011/11/12 03:23:40 UTC
Consumer state
" rather than store metadata for each message (marking it as consumed,
say), we just need to store the "high water mark" for each combination
of consumer, topic, and partition"
Typically where would one store this offset?
Re: Consumer state
Posted by Neha Narkhede <ne...@gmail.com>.
Yes. The one Jay was talking about is the ConsumerConnector. I believe the
ruby client does not have that functionality.
Thanks,
Neha
On Saturday, November 12, 2011, Mark <st...@gmail.com> wrote:
> So this is handled automatically via the java consumer client? Didn't
realize since I am mainly looking into the ruby client.
>
> On 11/11/11 7:09 PM, Jay Kreps wrote:
>>
>> The consumer stores the offset for you in zookeeper. This occurs
>> periodically in the background (if you enable auto.commit=true), or when
>> you call commit() explicitly.
>>
>> -Jay
>>
>> On Fri, Nov 11, 2011 at 6:23 PM, Mark<st...@gmail.com> wrote:
>>
>>> " rather than store metadata for each message (marking it as consumed,
>>> say), we just need to store the "high water mark" for each combination
of
>>> consumer, topic, and partition"
>>>
>>> Typically where would one store this offset?
>>>
>>>
>>>
>
Re: Consumer state
Posted by Mark <st...@gmail.com>.
So this is handled automatically via the java consumer client? Didn't
realize since I am mainly looking into the ruby client.
On 11/11/11 7:09 PM, Jay Kreps wrote:
> The consumer stores the offset for you in zookeeper. This occurs
> periodically in the background (if you enable auto.commit=true), or when
> you call commit() explicitly.
>
> -Jay
>
> On Fri, Nov 11, 2011 at 6:23 PM, Mark<st...@gmail.com> wrote:
>
>> " rather than store metadata for each message (marking it as consumed,
>> say), we just need to store the "high water mark" for each combination of
>> consumer, topic, and partition"
>>
>> Typically where would one store this offset?
>>
>>
>>
Re: Consumer state
Posted by Jay Kreps <ja...@gmail.com>.
The consumer stores the offset for you in zookeeper. This occurs
periodically in the background (if you enable auto.commit=true), or when
you call commit() explicitly.
-Jay
On Fri, Nov 11, 2011 at 6:23 PM, Mark <st...@gmail.com> wrote:
> " rather than store metadata for each message (marking it as consumed,
> say), we just need to store the "high water mark" for each combination of
> consumer, topic, and partition"
>
> Typically where would one store this offset?
>
>
>