You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark Grabois <ma...@trendrr.com> on 2012/05/07 05:36:52 UTC
consumer offset reset use case
Hello,
I know the topic of consumer offsets has come up frequently on the list,
how it is not possible to control the offset without using SimpleConsumer.
I would like to be able to reset all consumer offsets to "largest" (so that
consumption may start with the latest produced messages) when consumption
begins to lag behind our data feed. Is there a good way to do it with
zookeeper (since we will be using multiple consumers), without restarting
consumer?
Thank you,
Mark
Re: consumer offset reset use case
Posted by Jun Rao <ju...@gmail.com>.
Mark,
Currently, there is no reset API in the consumer. We are thinking about how
to add that properly in the future. For your use case, you can potentially
increase the degree of parallelism in the consumer to drive up the
consumption rate or throttle the consumption in your app.
Thanks,
Jun
On Sun, May 6, 2012 at 8:36 PM, Mark Grabois <ma...@trendrr.com>wrote:
> Hello,
>
> I know the topic of consumer offsets has come up frequently on the list,
> how it is not possible to control the offset without using SimpleConsumer.
> I would like to be able to reset all consumer offsets to "largest" (so that
> consumption may start with the latest produced messages) when consumption
> begins to lag behind our data feed. Is there a good way to do it with
> zookeeper (since we will be using multiple consumers), without restarting
> consumer?
>
> Thank you,
> Mark
>
Re: consumer offset reset use case
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
Hi Evan,
Exactly! We're deprecating our regular Scala actors for Akka's now
that 2.x is out in the wild (=
hmb.
On Mon, May 14, 2012 at 4:02 AM, Evan Chan <ev...@ooyala.com> wrote:
> Hisham,
>
> Thanks.
> A similar concept is Akka Agents (ever since discovering Akka, I haven't
> gone back to using the built in Actors library):
> http://doc.akka.io/docs/akka/2.0.1/scala/agents.html
>
> -Evan
>
>
> On Fri, May 11, 2012 at 6:54 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
>
>> Evan,
>>
>> AsyncValue is here:
>>
>> https://gist.github.com/2663670
>>
>> I'm using the ZK API from org.apache.zookeeper.ZooKeeper.
>>
>> hmb.
>>
>> On Thu, May 10, 2012 at 1:00 PM, Evan Chan <ev...@ooyala.com> wrote:
>> > Hisham,
>> >
>> > Thanks, what ZK client are you using, and is it possible to share
>> > AsyncValue as well?
>> >
>> > -Evan
>> >
>> >
>> > On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <hisham@mate1inc.com
>> >wrote:
>> >
>> >> Mark,
>> >>
>> >> You'll find some of the code here:
>> >>
>> >> https://gist.github.com/2654382
>> >>
>> >> It uses AsyncValue which is a custom class but the rest is pretty
>> >> simple to modify and use for your own purposes.
>> >>
>> >> Hope this helps.
>> >>
>> >> hmb.
>> >>
>> >> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <
>> mark.grabois@trendrr.com>
>> >> wrote:
>> >> > If you could that'd be great, thanks!
>> >> >
>> >> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <
>> hisham@mate1inc.com
>> >> >wrote:
>> >> >
>> >> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
>> mark.grabois@trendrr.com>
>> >> >> wrote:
>> >> >> > Hisham, thanks for the advice. Can you elaborate on how you moved
>> the
>> >> >> > offset in ZK? I read in the archives some people cleared the
>> offsets
>> >> >> > altogether. Did you access the ZK filesystem location where the
>> >> offsets
>> >> >> are
>> >> >> > stored:
>> >> >> >
>> >> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> >> >> > offset_counter_value ((persistent node)
>> >> >>
>> >> >> Mark,
>> >> >>
>> >> >> I've tried both approaches:
>> >> >>
>> >> >> - deleted the ZK nodes that hold the offsets
>> >> >> - reset the value of those nodes to zero
>> >> >>
>> >> >> Both worked however I stuck with deleting the nodes (since the files
>> >> >> represent broker/partition). I can share some of the code if you'd
>> >> >> like.
>> >> >>
>> >> >> hmb.
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
>> >> hisham@mate1inc.com
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Mark,
>> >> >> >>
>> >> >> >> We have similar needs and have added calls to our code to stop
>> >> >> >> consumption, move the offset in ZK, then resume (we use it to go
>> back
>> >> >> >> in time).
>> >> >> >>
>> >> >> >> hmb.
>> >> >> >>
>> >> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
>> >> mark.grabois@trendrr.com
>> >> >> >
>> >> >> >> wrote:
>> >> >> >> > Hello,
>> >> >> >> >
>> >> >> >> > I know the topic of consumer offsets has come up frequently on
>> the
>> >> >> list,
>> >> >> >> > how it is not possible to control the offset without using
>> >> >> >> SimpleConsumer.
>> >> >> >> > I would like to be able to reset all consumer offsets to
>> "largest"
>> >> (so
>> >> >> >> that
>> >> >> >> > consumption may start with the latest produced messages) when
>> >> >> consumption
>> >> >> >> > begins to lag behind our data feed. Is there a good way to do it
>> >> with
>> >> >> >> > zookeeper (since we will be using multiple consumers), without
>> >> >> restarting
>> >> >> >> > consumer?
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Mark
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Hisham Mardam-Bey
>> >> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >> >> >>
>> >> >> >> A: Because it messes up the order in which people normally read
>> text.
>> >> >> >> Q: Why is top-posting such a bad thing?
>> >> >> >> A: Top-posting.
>> >> >> >> Q: What is the most annoying thing in e-mail?
>> >> >> >>
>> >> >> >> -=[ Codito Ergo Sum ]=-
>> >> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Hisham Mardam-Bey
>> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >> >>
>> >> >> A: Because it messes up the order in which people normally read text.
>> >> >> Q: Why is top-posting such a bad thing?
>> >> >> A: Top-posting.
>> >> >> Q: What is the most annoying thing in e-mail?
>> >> >>
>> >> >> -=[ Codito Ergo Sum ]=-
>> >> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Hisham Mardam-Bey
>> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >>
>> >> A: Because it messes up the order in which people normally read text.
>> >> Q: Why is top-posting such a bad thing?
>> >> A: Top-posting.
>> >> Q: What is the most annoying thing in e-mail?
>> >>
>> >> -=[ Codito Ergo Sum ]=-
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > *Evan Chan*
>> > Senior Software Engineer |
>> > ev@ooyala.com | (650) 996-4600
>> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
>> > @ooyala<http://www.twitter.com/ooyala>
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-
Re: consumer offset reset use case
Posted by Evan Chan <ev...@ooyala.com>.
Hisham,
Thanks.
A similar concept is Akka Agents (ever since discovering Akka, I haven't
gone back to using the built in Actors library):
http://doc.akka.io/docs/akka/2.0.1/scala/agents.html
-Evan
On Fri, May 11, 2012 at 6:54 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
> Evan,
>
> AsyncValue is here:
>
> https://gist.github.com/2663670
>
> I'm using the ZK API from org.apache.zookeeper.ZooKeeper.
>
> hmb.
>
> On Thu, May 10, 2012 at 1:00 PM, Evan Chan <ev...@ooyala.com> wrote:
> > Hisham,
> >
> > Thanks, what ZK client are you using, and is it possible to share
> > AsyncValue as well?
> >
> > -Evan
> >
> >
> > On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
> >
> >> Mark,
> >>
> >> You'll find some of the code here:
> >>
> >> https://gist.github.com/2654382
> >>
> >> It uses AsyncValue which is a custom class but the rest is pretty
> >> simple to modify and use for your own purposes.
> >>
> >> Hope this helps.
> >>
> >> hmb.
> >>
> >> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <
> mark.grabois@trendrr.com>
> >> wrote:
> >> > If you could that'd be great, thanks!
> >> >
> >> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <
> hisham@mate1inc.com
> >> >wrote:
> >> >
> >> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> mark.grabois@trendrr.com>
> >> >> wrote:
> >> >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> >> >> > offset in ZK? I read in the archives some people cleared the
> offsets
> >> >> > altogether. Did you access the ZK filesystem location where the
> >> offsets
> >> >> are
> >> >> > stored:
> >> >> >
> >> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> >> >> > offset_counter_value ((persistent node)
> >> >>
> >> >> Mark,
> >> >>
> >> >> I've tried both approaches:
> >> >>
> >> >> - deleted the ZK nodes that hold the offsets
> >> >> - reset the value of those nodes to zero
> >> >>
> >> >> Both worked however I stuck with deleting the nodes (since the files
> >> >> represent broker/partition). I can share some of the code if you'd
> >> >> like.
> >> >>
> >> >> hmb.
> >> >>
> >> >> >
> >> >> >
> >> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> >> hisham@mate1inc.com
> >> >> >wrote:
> >> >> >
> >> >> >> Mark,
> >> >> >>
> >> >> >> We have similar needs and have added calls to our code to stop
> >> >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> >> >> >> in time).
> >> >> >>
> >> >> >> hmb.
> >> >> >>
> >> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> >> mark.grabois@trendrr.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hello,
> >> >> >> >
> >> >> >> > I know the topic of consumer offsets has come up frequently on
> the
> >> >> list,
> >> >> >> > how it is not possible to control the offset without using
> >> >> >> SimpleConsumer.
> >> >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> >> (so
> >> >> >> that
> >> >> >> > consumption may start with the latest produced messages) when
> >> >> consumption
> >> >> >> > begins to lag behind our data feed. Is there a good way to do it
> >> with
> >> >> >> > zookeeper (since we will be using multiple consumers), without
> >> >> restarting
> >> >> >> > consumer?
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Mark
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Hisham Mardam-Bey
> >> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >> >>
> >> >> >> A: Because it messes up the order in which people normally read
> text.
> >> >> >> Q: Why is top-posting such a bad thing?
> >> >> >> A: Top-posting.
> >> >> >> Q: What is the most annoying thing in e-mail?
> >> >> >>
> >> >> >> -=[ Codito Ergo Sum ]=-
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Hisham Mardam-Bey
> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >>
> >> >> A: Because it messes up the order in which people normally read text.
> >> >> Q: Why is top-posting such a bad thing?
> >> >> A: Top-posting.
> >> >> Q: What is the most annoying thing in e-mail?
> >> >>
> >> >> -=[ Codito Ergo Sum ]=-
> >> >>
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >>
> >> A: Because it messes up the order in which people normally read text.
> >> Q: Why is top-posting such a bad thing?
> >> A: Top-posting.
> >> Q: What is the most annoying thing in e-mail?
> >>
> >> -=[ Codito Ergo Sum ]=-
> >>
> >
> >
> >
> > --
> > --
> > *Evan Chan*
> > Senior Software Engineer |
> > ev@ooyala.com | (650) 996-4600
> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > @ooyala<http://www.twitter.com/ooyala>
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
--
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>
Re: consumer offset reset use case
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
Evan,
AsyncValue is here:
https://gist.github.com/2663670
I'm using the ZK API from org.apache.zookeeper.ZooKeeper.
hmb.
On Thu, May 10, 2012 at 1:00 PM, Evan Chan <ev...@ooyala.com> wrote:
> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
>
>> Mark,
>>
>> You'll find some of the code here:
>>
>> https://gist.github.com/2654382
>>
>> It uses AsyncValue which is a custom class but the rest is pretty
>> simple to modify and use for your own purposes.
>>
>> Hope this helps.
>>
>> hmb.
>>
>> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <ma...@trendrr.com>
>> wrote:
>> > If you could that'd be great, thanks!
>> >
>> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <hisham@mate1inc.com
>> >wrote:
>> >
>> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <ma...@trendrr.com>
>> >> wrote:
>> >> > Hisham, thanks for the advice. Can you elaborate on how you moved the
>> >> > offset in ZK? I read in the archives some people cleared the offsets
>> >> > altogether. Did you access the ZK filesystem location where the
>> offsets
>> >> are
>> >> > stored:
>> >> >
>> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> >> > offset_counter_value ((persistent node)
>> >>
>> >> Mark,
>> >>
>> >> I've tried both approaches:
>> >>
>> >> - deleted the ZK nodes that hold the offsets
>> >> - reset the value of those nodes to zero
>> >>
>> >> Both worked however I stuck with deleting the nodes (since the files
>> >> represent broker/partition). I can share some of the code if you'd
>> >> like.
>> >>
>> >> hmb.
>> >>
>> >> >
>> >> >
>> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
>> hisham@mate1inc.com
>> >> >wrote:
>> >> >
>> >> >> Mark,
>> >> >>
>> >> >> We have similar needs and have added calls to our code to stop
>> >> >> consumption, move the offset in ZK, then resume (we use it to go back
>> >> >> in time).
>> >> >>
>> >> >> hmb.
>> >> >>
>> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
>> mark.grabois@trendrr.com
>> >> >
>> >> >> wrote:
>> >> >> > Hello,
>> >> >> >
>> >> >> > I know the topic of consumer offsets has come up frequently on the
>> >> list,
>> >> >> > how it is not possible to control the offset without using
>> >> >> SimpleConsumer.
>> >> >> > I would like to be able to reset all consumer offsets to "largest"
>> (so
>> >> >> that
>> >> >> > consumption may start with the latest produced messages) when
>> >> consumption
>> >> >> > begins to lag behind our data feed. Is there a good way to do it
>> with
>> >> >> > zookeeper (since we will be using multiple consumers), without
>> >> restarting
>> >> >> > consumer?
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Mark
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Hisham Mardam-Bey
>> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >> >>
>> >> >> A: Because it messes up the order in which people normally read text.
>> >> >> Q: Why is top-posting such a bad thing?
>> >> >> A: Top-posting.
>> >> >> Q: What is the most annoying thing in e-mail?
>> >> >>
>> >> >> -=[ Codito Ergo Sum ]=-
>> >> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Hisham Mardam-Bey
>> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >>
>> >> A: Because it messes up the order in which people normally read text.
>> >> Q: Why is top-posting such a bad thing?
>> >> A: Top-posting.
>> >> Q: What is the most annoying thing in e-mail?
>> >>
>> >> -=[ Codito Ergo Sum ]=-
>> >>
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-
Re: consumer offset reset use case
Posted by Felix GV <fe...@mate1inc.com>.
You should be able to replace AsyncValue[Boolean] with an
AtomicBoolean<http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/atomic/AtomicBoolean.html>
.
As for the ZK client, maybe I don't understand your question correctly, but
I think this code is simply relying on the Zookeeper class it would find in
a standard ZK jar.
--
Felix
On Thu, May 10, 2012 at 1:00 PM, Evan Chan <ev...@ooyala.com> wrote:
> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
>
> > Mark,
> >
> > You'll find some of the code here:
> >
> > https://gist.github.com/2654382
> >
> > It uses AsyncValue which is a custom class but the rest is pretty
> > simple to modify and use for your own purposes.
> >
> > Hope this helps.
> >
> > hmb.
> >
> > On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <mark.grabois@trendrr.com
> >
> > wrote:
> > > If you could that'd be great, thanks!
> > >
> > > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <hisham@mate1inc.com
> > >wrote:
> > >
> > >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> mark.grabois@trendrr.com>
> > >> wrote:
> > >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> > >> > offset in ZK? I read in the archives some people cleared the offsets
> > >> > altogether. Did you access the ZK filesystem location where the
> > offsets
> > >> are
> > >> > stored:
> > >> >
> > >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > >> > offset_counter_value ((persistent node)
> > >>
> > >> Mark,
> > >>
> > >> I've tried both approaches:
> > >>
> > >> - deleted the ZK nodes that hold the offsets
> > >> - reset the value of those nodes to zero
> > >>
> > >> Both worked however I stuck with deleting the nodes (since the files
> > >> represent broker/partition). I can share some of the code if you'd
> > >> like.
> > >>
> > >> hmb.
> > >>
> > >> >
> > >> >
> > >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> > hisham@mate1inc.com
> > >> >wrote:
> > >> >
> > >> >> Mark,
> > >> >>
> > >> >> We have similar needs and have added calls to our code to stop
> > >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> > >> >> in time).
> > >> >>
> > >> >> hmb.
> > >> >>
> > >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> > mark.grabois@trendrr.com
> > >> >
> > >> >> wrote:
> > >> >> > Hello,
> > >> >> >
> > >> >> > I know the topic of consumer offsets has come up frequently on
> the
> > >> list,
> > >> >> > how it is not possible to control the offset without using
> > >> >> SimpleConsumer.
> > >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> > (so
> > >> >> that
> > >> >> > consumption may start with the latest produced messages) when
> > >> consumption
> > >> >> > begins to lag behind our data feed. Is there a good way to do it
> > with
> > >> >> > zookeeper (since we will be using multiple consumers), without
> > >> restarting
> > >> >> > consumer?
> > >> >> >
> > >> >> > Thank you,
> > >> >> > Mark
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Hisham Mardam-Bey
> > >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >> >>
> > >> >> A: Because it messes up the order in which people normally read
> text.
> > >> >> Q: Why is top-posting such a bad thing?
> > >> >> A: Top-posting.
> > >> >> Q: What is the most annoying thing in e-mail?
> > >> >>
> > >> >> -=[ Codito Ergo Sum ]=-
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Hisham Mardam-Bey
> > >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >>
> > >> A: Because it messes up the order in which people normally read text.
> > >> Q: Why is top-posting such a bad thing?
> > >> A: Top-posting.
> > >> Q: What is the most annoying thing in e-mail?
> > >>
> > >> -=[ Codito Ergo Sum ]=-
> > >>
> >
> >
> >
> > --
> > Hisham Mardam-Bey
> > [ Director of Engineering ] [ Mate1 Inc. ]
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>
Re: consumer offset reset use case
Posted by Evan Chan <ev...@ooyala.com>.
Hisham,
Thanks, what ZK client are you using, and is it possible to share
AsyncValue as well?
-Evan
On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
> Mark,
>
> You'll find some of the code here:
>
> https://gist.github.com/2654382
>
> It uses AsyncValue which is a custom class but the rest is pretty
> simple to modify and use for your own purposes.
>
> Hope this helps.
>
> hmb.
>
> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <ma...@trendrr.com>
> wrote:
> > If you could that'd be great, thanks!
> >
> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
> >
> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <ma...@trendrr.com>
> >> wrote:
> >> > Hisham, thanks for the advice. Can you elaborate on how you moved the
> >> > offset in ZK? I read in the archives some people cleared the offsets
> >> > altogether. Did you access the ZK filesystem location where the
> offsets
> >> are
> >> > stored:
> >> >
> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> >> > offset_counter_value ((persistent node)
> >>
> >> Mark,
> >>
> >> I've tried both approaches:
> >>
> >> - deleted the ZK nodes that hold the offsets
> >> - reset the value of those nodes to zero
> >>
> >> Both worked however I stuck with deleting the nodes (since the files
> >> represent broker/partition). I can share some of the code if you'd
> >> like.
> >>
> >> hmb.
> >>
> >> >
> >> >
> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> hisham@mate1inc.com
> >> >wrote:
> >> >
> >> >> Mark,
> >> >>
> >> >> We have similar needs and have added calls to our code to stop
> >> >> consumption, move the offset in ZK, then resume (we use it to go back
> >> >> in time).
> >> >>
> >> >> hmb.
> >> >>
> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> mark.grabois@trendrr.com
> >> >
> >> >> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I know the topic of consumer offsets has come up frequently on the
> >> list,
> >> >> > how it is not possible to control the offset without using
> >> >> SimpleConsumer.
> >> >> > I would like to be able to reset all consumer offsets to "largest"
> (so
> >> >> that
> >> >> > consumption may start with the latest produced messages) when
> >> consumption
> >> >> > begins to lag behind our data feed. Is there a good way to do it
> with
> >> >> > zookeeper (since we will be using multiple consumers), without
> >> restarting
> >> >> > consumer?
> >> >> >
> >> >> > Thank you,
> >> >> > Mark
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Hisham Mardam-Bey
> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >>
> >> >> A: Because it messes up the order in which people normally read text.
> >> >> Q: Why is top-posting such a bad thing?
> >> >> A: Top-posting.
> >> >> Q: What is the most annoying thing in e-mail?
> >> >>
> >> >> -=[ Codito Ergo Sum ]=-
> >> >>
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >>
> >> A: Because it messes up the order in which people normally read text.
> >> Q: Why is top-posting such a bad thing?
> >> A: Top-posting.
> >> Q: What is the most annoying thing in e-mail?
> >>
> >> -=[ Codito Ergo Sum ]=-
> >>
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
--
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>
Re: consumer offset reset use case
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
Mark,
You'll find some of the code here:
https://gist.github.com/2654382
It uses AsyncValue which is a custom class but the rest is pretty
simple to modify and use for your own purposes.
Hope this helps.
hmb.
On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <ma...@trendrr.com> wrote:
> If you could that'd be great, thanks!
>
> On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
>
>> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <ma...@trendrr.com>
>> wrote:
>> > Hisham, thanks for the advice. Can you elaborate on how you moved the
>> > offset in ZK? I read in the archives some people cleared the offsets
>> > altogether. Did you access the ZK filesystem location where the offsets
>> are
>> > stored:
>> >
>> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> > offset_counter_value ((persistent node)
>>
>> Mark,
>>
>> I've tried both approaches:
>>
>> - deleted the ZK nodes that hold the offsets
>> - reset the value of those nodes to zero
>>
>> Both worked however I stuck with deleting the nodes (since the files
>> represent broker/partition). I can share some of the code if you'd
>> like.
>>
>> hmb.
>>
>> >
>> >
>> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <hisham@mate1inc.com
>> >wrote:
>> >
>> >> Mark,
>> >>
>> >> We have similar needs and have added calls to our code to stop
>> >> consumption, move the offset in ZK, then resume (we use it to go back
>> >> in time).
>> >>
>> >> hmb.
>> >>
>> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <mark.grabois@trendrr.com
>> >
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > I know the topic of consumer offsets has come up frequently on the
>> list,
>> >> > how it is not possible to control the offset without using
>> >> SimpleConsumer.
>> >> > I would like to be able to reset all consumer offsets to "largest" (so
>> >> that
>> >> > consumption may start with the latest produced messages) when
>> consumption
>> >> > begins to lag behind our data feed. Is there a good way to do it with
>> >> > zookeeper (since we will be using multiple consumers), without
>> restarting
>> >> > consumer?
>> >> >
>> >> > Thank you,
>> >> > Mark
>> >>
>> >>
>> >>
>> >> --
>> >> Hisham Mardam-Bey
>> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >>
>> >> A: Because it messes up the order in which people normally read text.
>> >> Q: Why is top-posting such a bad thing?
>> >> A: Top-posting.
>> >> Q: What is the most annoying thing in e-mail?
>> >>
>> >> -=[ Codito Ergo Sum ]=-
>> >>
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>
--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-
Re: consumer offset reset use case
Posted by Mark Grabois <ma...@trendrr.com>.
If you could that'd be great, thanks!
On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <ma...@trendrr.com>
> wrote:
> > Hisham, thanks for the advice. Can you elaborate on how you moved the
> > offset in ZK? I read in the archives some people cleared the offsets
> > altogether. Did you access the ZK filesystem location where the offsets
> are
> > stored:
> >
> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > offset_counter_value ((persistent node)
>
> Mark,
>
> I've tried both approaches:
>
> - deleted the ZK nodes that hold the offsets
> - reset the value of those nodes to zero
>
> Both worked however I stuck with deleting the nodes (since the files
> represent broker/partition). I can share some of the code if you'd
> like.
>
> hmb.
>
> >
> >
> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
> >
> >> Mark,
> >>
> >> We have similar needs and have added calls to our code to stop
> >> consumption, move the offset in ZK, then resume (we use it to go back
> >> in time).
> >>
> >> hmb.
> >>
> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <mark.grabois@trendrr.com
> >
> >> wrote:
> >> > Hello,
> >> >
> >> > I know the topic of consumer offsets has come up frequently on the
> list,
> >> > how it is not possible to control the offset without using
> >> SimpleConsumer.
> >> > I would like to be able to reset all consumer offsets to "largest" (so
> >> that
> >> > consumption may start with the latest produced messages) when
> consumption
> >> > begins to lag behind our data feed. Is there a good way to do it with
> >> > zookeeper (since we will be using multiple consumers), without
> restarting
> >> > consumer?
> >> >
> >> > Thank you,
> >> > Mark
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >>
> >> A: Because it messes up the order in which people normally read text.
> >> Q: Why is top-posting such a bad thing?
> >> A: Top-posting.
> >> Q: What is the most annoying thing in e-mail?
> >>
> >> -=[ Codito Ergo Sum ]=-
> >>
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
Re: consumer offset reset use case
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <ma...@trendrr.com> wrote:
> Hisham, thanks for the advice. Can you elaborate on how you moved the
> offset in ZK? I read in the archives some people cleared the offsets
> altogether. Did you access the ZK filesystem location where the offsets are
> stored:
>
> /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> offset_counter_value ((persistent node)
Mark,
I've tried both approaches:
- deleted the ZK nodes that hold the offsets
- reset the value of those nodes to zero
Both worked however I stuck with deleting the nodes (since the files
represent broker/partition). I can share some of the code if you'd
like.
hmb.
>
>
> On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
>
>> Mark,
>>
>> We have similar needs and have added calls to our code to stop
>> consumption, move the offset in ZK, then resume (we use it to go back
>> in time).
>>
>> hmb.
>>
>> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <ma...@trendrr.com>
>> wrote:
>> > Hello,
>> >
>> > I know the topic of consumer offsets has come up frequently on the list,
>> > how it is not possible to control the offset without using
>> SimpleConsumer.
>> > I would like to be able to reset all consumer offsets to "largest" (so
>> that
>> > consumption may start with the latest produced messages) when consumption
>> > begins to lag behind our data feed. Is there a good way to do it with
>> > zookeeper (since we will be using multiple consumers), without restarting
>> > consumer?
>> >
>> > Thank you,
>> > Mark
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>
--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-
Re: consumer offset reset use case
Posted by Mark Grabois <ma...@trendrr.com>.
Hisham, thanks for the advice. Can you elaborate on how you moved the
offset in ZK? I read in the archives some people cleared the offsets
altogether. Did you access the ZK filesystem location where the offsets are
stored:
/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
offset_counter_value ((persistent node)
On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
> Mark,
>
> We have similar needs and have added calls to our code to stop
> consumption, move the offset in ZK, then resume (we use it to go back
> in time).
>
> hmb.
>
> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <ma...@trendrr.com>
> wrote:
> > Hello,
> >
> > I know the topic of consumer offsets has come up frequently on the list,
> > how it is not possible to control the offset without using
> SimpleConsumer.
> > I would like to be able to reset all consumer offsets to "largest" (so
> that
> > consumption may start with the latest produced messages) when consumption
> > begins to lag behind our data feed. Is there a good way to do it with
> > zookeeper (since we will be using multiple consumers), without restarting
> > consumer?
> >
> > Thank you,
> > Mark
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
Re: consumer offset reset use case
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
Mark,
We have similar needs and have added calls to our code to stop
consumption, move the offset in ZK, then resume (we use it to go back
in time).
hmb.
On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <ma...@trendrr.com> wrote:
> Hello,
>
> I know the topic of consumer offsets has come up frequently on the list,
> how it is not possible to control the offset without using SimpleConsumer.
> I would like to be able to reset all consumer offsets to "largest" (so that
> consumption may start with the latest produced messages) when consumption
> begins to lag behind our data feed. Is there a good way to do it with
> zookeeper (since we will be using multiple consumers), without restarting
> consumer?
>
> Thank you,
> Mark
--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-