You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Michael Luban <mi...@gmail.com> on 2012/01/24 01:12:32 UTC
Specifying initial message offset with ZookeeperConnectorConsumer
I'm interested in having my consumer service start consuming the most
recent message rather than the message corresponding to the last offset
stored. For instance, if the kafka consumer service is down for a period
of time, I would like to skip over all messages since the last recorded
offset and begin consuming from the most recent message.
According to the following, it seems I have to use SimpleConsumer to
achieve this? Seems like this would be difficult to manage in a "load
balanced" consumer scenario (federation of consumers in the same group
consuming the same topic).
https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76
Any additional insight appreciated.
Re: Specifying initial message offset with ZookeeperConnectorConsumer
Posted by Michael Luban <mi...@gmail.com>.
autooffset.reset="largest" in conjunction with deleting the corresponding
zookeeper offsets solved the problem. Thanks to both for sharing this
excellent advice.
On Mon, Jan 23, 2012 at 8:12 PM, Joel Koshy <jj...@gmail.com> wrote:
> Largest works - however, autooffset only applies if the consumer group
> is new (non-existent in ZK) or if the stored offset is out of range.
> If you restart your consumer at a later point it will resume from the
> last check-pointed offset. So from the scenario you describe (down for
> a period of time...) it appears that it wouldn't be enough for you.
> Depending on the details of your set up, one thing you could do is to
> clean up the zookeeper entries for your consumer group before bringing
> up your consumer service again.
>
> Joel
>
> On Mon, Jan 23, 2012 at 4:49 PM, Evan Chan <ev...@ooyala.com> wrote:
> > Michael,
> >
> > It seems there is a property "autooffset.reset" here:
> > http://incubator.apache.org/kafka/configuration.html
> >
> > which you can set to "largest". I haven't used it though so I can't be
> > sure.
> >
> > -Evan
> >
> >
> > On Mon, Jan 23, 2012 at 4:12 PM, Michael Luban <michael.luban@gmail.com
> >wrote:
> >
> >> I'm interested in having my consumer service start consuming the most
> >> recent message rather than the message corresponding to the last offset
> >> stored. For instance, if the kafka consumer service is down for a
> period
> >> of time, I would like to skip over all messages since the last recorded
> >> offset and begin consuming from the most recent message.
> >>
> >> According to the following, it seems I have to use SimpleConsumer to
> >> achieve this? Seems like this would be difficult to manage in a "load
> >> balanced" consumer scenario (federation of consumers in the same group
> >> consuming the same topic).
> >>
> >>
> https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76
> >>
> >> Any additional insight appreciated.
> >>
> >
> >
> >
> > --
> > --
> > *Evan Chan*
> > Senior Software Engineer |
> > ev@ooyala.com | (650) 996-4600
> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > @ooyala<http://www.twitter.com/ooyala>
>
Re: Specifying initial message offset with ZookeeperConnectorConsumer
Posted by Joel Koshy <jj...@gmail.com>.
Largest works - however, autooffset only applies if the consumer group
is new (non-existent in ZK) or if the stored offset is out of range.
If you restart your consumer at a later point it will resume from the
last check-pointed offset. So from the scenario you describe (down for
a period of time...) it appears that it wouldn't be enough for you.
Depending on the details of your set up, one thing you could do is to
clean up the zookeeper entries for your consumer group before bringing
up your consumer service again.
Joel
On Mon, Jan 23, 2012 at 4:49 PM, Evan Chan <ev...@ooyala.com> wrote:
> Michael,
>
> It seems there is a property "autooffset.reset" here:
> http://incubator.apache.org/kafka/configuration.html
>
> which you can set to "largest". I haven't used it though so I can't be
> sure.
>
> -Evan
>
>
> On Mon, Jan 23, 2012 at 4:12 PM, Michael Luban <mi...@gmail.com>wrote:
>
>> I'm interested in having my consumer service start consuming the most
>> recent message rather than the message corresponding to the last offset
>> stored. For instance, if the kafka consumer service is down for a period
>> of time, I would like to skip over all messages since the last recorded
>> offset and begin consuming from the most recent message.
>>
>> According to the following, it seems I have to use SimpleConsumer to
>> achieve this? Seems like this would be difficult to manage in a "load
>> balanced" consumer scenario (federation of consumers in the same group
>> consuming the same topic).
>>
>> https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76
>>
>> Any additional insight appreciated.
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
Re: Specifying initial message offset with ZookeeperConnectorConsumer
Posted by Evan Chan <ev...@ooyala.com>.
Michael,
It seems there is a property "autooffset.reset" here:
http://incubator.apache.org/kafka/configuration.html
which you can set to "largest". I haven't used it though so I can't be
sure.
-Evan
On Mon, Jan 23, 2012 at 4:12 PM, Michael Luban <mi...@gmail.com>wrote:
> I'm interested in having my consumer service start consuming the most
> recent message rather than the message corresponding to the last offset
> stored. For instance, if the kafka consumer service is down for a period
> of time, I would like to skip over all messages since the last recorded
> offset and begin consuming from the most recent message.
>
> According to the following, it seems I have to use SimpleConsumer to
> achieve this? Seems like this would be difficult to manage in a "load
> balanced" consumer scenario (federation of consumers in the same group
> consuming the same topic).
>
> https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76
>
> Any additional insight appreciated.
>
--
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>