You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Jain, Kokil" <Ja...@bit-sys.com> on 2012/03/12 20:09:39 UTC
Replay
Hi,
Sorry, new to Kafka. Is it possible for a consumer to replay messages from
an "offset" that may have been committed before?
Thanks for the help,
Kokil Jain
Re: Replay
Posted by Jun Rao <ju...@gmail.com>.
There is a tool ImportZkOffsets that allows you to import offsets (assuming
they are valid) to ZK. However, it's an admin tool and you need to stop the
consumer before using the tool.
Thanks,
Jun
On Thu, Mar 29, 2012 at 10:16 AM, Neha Narkhede <ne...@gmail.com>wrote:
> Elben,
>
> What I had suggested was the following -
>
> >> Or sensei can provide a restart from tail API, that will delete the
> offset state from ZK and restart its Kafka consumer. Currently that is the
> best way to have a Kafka consumer restart fetching latest data.
>
> Note that sensei would need to first get rid of the current state for
> the consumer group from ZK (delete /consumers/[your-consumer-group]).
> And then when you restart the Kafka consumer, it will start consuming
> from the latest data available on the server.
>
> Thanks,
> Neha
>
> On Thu, Mar 29, 2012 at 10:10 AM, Jun Rao <ju...@gmail.com> wrote:
> > Elben,
> >
> > autooffset.reset="largest" is only used for the very first time the
> > consumer is started when there is no offset registered in ZK. Once the
> > consumer is started, by default it periodically checkpoints offset in ZK
> > and on subsequent restarts, the consumer starts consumption from the
> offset
> > stored in ZK.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <el...@gmail.com>
> wrote:
> >
> >> Hi Neha,
> >>
> >> What does it mean to fetch the latest data? If there are items in the
> kafka
> >> queue and no new items are coming in, would autooffset.reset="largest"
> wait
> >> for new items or would it consume some of the existing items in the
> queue?
> >>
> >> I'm asking this because I've modified the sensei gateways
> >> to autooffset.reset="largest" but it's still consuming items previously
> in
> >> the queue, and I'm not sure if the problem is on my end.
> >>
> >> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find
> the
> >> latest offset for the queue itself (not the consumer)?
> >>
> >> Elben
> >>
> >>
> >> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >> >wrote:
> >>
> >> > John,
> >> >
> >> > You can achieve that via the autooffset.reset="largest" config option
> >> > in the consumer.
> >> >
> >> > Thanks,
> >> > Neha
> >> >
> >> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com>
> wrote:
> >> > > Hi Neha:
> >> > >
> >> > > Can you point me to the api to always get data from tail?
> >> > >
> >> > > Thanks
> >> > >
> >> > > -John
> >> > >
> >> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
> >> neha.narkhede@gmail.com
> >> > >wrote:
> >> > >
> >> > >> Elben,
> >> > >>
> >> > >> Or sensei can provide a restart from tail API, that will delete the
> >> > offset
> >> > >> state from ZK and restart its Kafka consumer. Currently that is the
> >> best
> >> > >> way to have a Kafka consumer restart fetching latest data.
> >> > >>
> >> > >> Thanks,
> >> > >> Neha
> >> > >> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
> >> > >>
> >> > >> > I think in the newer kafka release, there is a tool to update the
> >> > version
> >> > >> > for a client.
> >> > >> >
> >> > >> > Jun, can you elaborate on that?
> >> > >> >
> >> > >> > Thanks
> >> > >> >
> >> > >> > -John
> >> > >> >
> >> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <
> elbenshira@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > We're using use kafka+sensei at our company and we'd love to be
> >> > able to
> >> > >> > > start at the end of the queue (latest entries). The use case is
> >> when
> >> > >> > sensei
> >> > >> > > goes down or has to be restarted, we only care about indexing
> the
> >> > >> latest
> >> > >> > > items.
> >> > >> > >
> >> > >> > > A workaround I guess is to stop sensei, manually change the
> high
> >> > >> > watermark
> >> > >> > > in kafka, then restart sensei.
> >> > >> > >
> >> > >> > > Elben
> >> > >> > >
> >> > >> > >
> >> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com>
> >> wrote:
> >> > >> > >
> >> > >> > > > Casey,
> >> > >> > > >
> >> > >> > > > You can't get offsets using zk-based consumer now. We are
> still
> >> > >> trying
> >> > >> > to
> >> > >> > > > figure about how to support that in the future.
> >> > >> > > >
> >> > >> > > > Thanks,
> >> > >> > > >
> >> > >> > > > Jun
> >> > >> > > >
> >> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> >> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
> >> > >> > > >
> >> > >> > > > > Hello,
> >> > >> > > > >
> >> > >> > > > > Just out of curiosity, how would you get the offsets when
> >> using
> >> > a
> >> > >> > > > zk-based
> >> > >> > > > > consumer? I don't see anything that seems to let you do
> this
> >> > >> easily.
> >> > >> > > > >
> >> > >> > > > > Casey
> >> > >> > > > >
> >> > >> > > > > ________________________________________
> >> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> >> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> >> > >> > > > > To: kafka-users@incubator.apache.org
> >> > >> > > > > Subject: Re: Replay
> >> > >> > > > >
> >> > >> > > > > With simple consumer, you can start from any valid offset
> that
> >> > you
> >> > >> > keep
> >> > >> > > > > track off.
> >> > >> > > > >
> >> > >> > > > > With the high level (zk-based) consumer: you will need to
> stop
> >> > all
> >> > >> > > > > consumers, and then update the offsets in zookeeper (see
> the
> >> > >> > > > > ImportZkOffsets tool in the tools package) with the offsets
> >> you
> >> > >> want
> >> > >> > to
> >> > >> > > > > replay from before restarting your consumer.
> >> > >> > > > >
> >> > >> > > > > Thanks,
> >> > >> > > > >
> >> > >> > > > > Joel
> >> > >> > > > >
> >> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
> >> > JainK@bit-sys.com>
> >> > >> > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi,
> >> > >> > > > > >
> >> > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to
> replay
> >> > >> > messages
> >> > >> > > > > from
> >> > >> > > > > > an "offset" that may have been committed before?
> >> > >> > > > > >
> >> > >> > > > > > Thanks for the help,
> >> > >> > > > > > Kokil Jain
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
Re: Replay
Posted by Neha Narkhede <ne...@gmail.com>.
Elben,
What I had suggested was the following -
>> Or sensei can provide a restart from tail API, that will delete the offset state from ZK and restart its Kafka consumer. Currently that is the best way to have a Kafka consumer restart fetching latest data.
Note that sensei would need to first get rid of the current state for
the consumer group from ZK (delete /consumers/[your-consumer-group]).
And then when you restart the Kafka consumer, it will start consuming
from the latest data available on the server.
Thanks,
Neha
On Thu, Mar 29, 2012 at 10:10 AM, Jun Rao <ju...@gmail.com> wrote:
> Elben,
>
> autooffset.reset="largest" is only used for the very first time the
> consumer is started when there is no offset registered in ZK. Once the
> consumer is started, by default it periodically checkpoints offset in ZK
> and on subsequent restarts, the consumer starts consumption from the offset
> stored in ZK.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <el...@gmail.com> wrote:
>
>> Hi Neha,
>>
>> What does it mean to fetch the latest data? If there are items in the kafka
>> queue and no new items are coming in, would autooffset.reset="largest" wait
>> for new items or would it consume some of the existing items in the queue?
>>
>> I'm asking this because I've modified the sensei gateways
>> to autooffset.reset="largest" but it's still consuming items previously in
>> the queue, and I'm not sure if the problem is on my end.
>>
>> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the
>> latest offset for the queue itself (not the consumer)?
>>
>> Elben
>>
>>
>> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
>> >wrote:
>>
>> > John,
>> >
>> > You can achieve that via the autooffset.reset="largest" config option
>> > in the consumer.
>> >
>> > Thanks,
>> > Neha
>> >
>> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com> wrote:
>> > > Hi Neha:
>> > >
>> > > Can you point me to the api to always get data from tail?
>> > >
>> > > Thanks
>> > >
>> > > -John
>> > >
>> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> > >wrote:
>> > >
>> > >> Elben,
>> > >>
>> > >> Or sensei can provide a restart from tail API, that will delete the
>> > offset
>> > >> state from ZK and restart its Kafka consumer. Currently that is the
>> best
>> > >> way to have a Kafka consumer restart fetching latest data.
>> > >>
>> > >> Thanks,
>> > >> Neha
>> > >> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
>> > >>
>> > >> > I think in the newer kafka release, there is a tool to update the
>> > version
>> > >> > for a client.
>> > >> >
>> > >> > Jun, can you elaborate on that?
>> > >> >
>> > >> > Thanks
>> > >> >
>> > >> > -John
>> > >> >
>> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > We're using use kafka+sensei at our company and we'd love to be
>> > able to
>> > >> > > start at the end of the queue (latest entries). The use case is
>> when
>> > >> > sensei
>> > >> > > goes down or has to be restarted, we only care about indexing the
>> > >> latest
>> > >> > > items.
>> > >> > >
>> > >> > > A workaround I guess is to stop sensei, manually change the high
>> > >> > watermark
>> > >> > > in kafka, then restart sensei.
>> > >> > >
>> > >> > > Elben
>> > >> > >
>> > >> > >
>> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com>
>> wrote:
>> > >> > >
>> > >> > > > Casey,
>> > >> > > >
>> > >> > > > You can't get offsets using zk-based consumer now. We are still
>> > >> trying
>> > >> > to
>> > >> > > > figure about how to support that in the future.
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > >
>> > >> > > > Jun
>> > >> > > >
>> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
>> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
>> > >> > > >
>> > >> > > > > Hello,
>> > >> > > > >
>> > >> > > > > Just out of curiosity, how would you get the offsets when
>> using
>> > a
>> > >> > > > zk-based
>> > >> > > > > consumer? I don't see anything that seems to let you do this
>> > >> easily.
>> > >> > > > >
>> > >> > > > > Casey
>> > >> > > > >
>> > >> > > > > ________________________________________
>> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
>> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
>> > >> > > > > To: kafka-users@incubator.apache.org
>> > >> > > > > Subject: Re: Replay
>> > >> > > > >
>> > >> > > > > With simple consumer, you can start from any valid offset that
>> > you
>> > >> > keep
>> > >> > > > > track off.
>> > >> > > > >
>> > >> > > > > With the high level (zk-based) consumer: you will need to stop
>> > all
>> > >> > > > > consumers, and then update the offsets in zookeeper (see the
>> > >> > > > > ImportZkOffsets tool in the tools package) with the offsets
>> you
>> > >> want
>> > >> > to
>> > >> > > > > replay from before restarting your consumer.
>> > >> > > > >
>> > >> > > > > Thanks,
>> > >> > > > >
>> > >> > > > > Joel
>> > >> > > > >
>> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
>> > JainK@bit-sys.com>
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
>> > >> > messages
>> > >> > > > > from
>> > >> > > > > > an "offset" that may have been committed before?
>> > >> > > > > >
>> > >> > > > > > Thanks for the help,
>> > >> > > > > > Kokil Jain
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
Re: Replay
Posted by Jun Rao <ju...@gmail.com>.
Elben,
autooffset.reset="largest" is only used for the very first time the
consumer is started when there is no offset registered in ZK. Once the
consumer is started, by default it periodically checkpoints offset in ZK
and on subsequent restarts, the consumer starts consumption from the offset
stored in ZK.
Thanks,
Jun
On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <el...@gmail.com> wrote:
> Hi Neha,
>
> What does it mean to fetch the latest data? If there are items in the kafka
> queue and no new items are coming in, would autooffset.reset="largest" wait
> for new items or would it consume some of the existing items in the queue?
>
> I'm asking this because I've modified the sensei gateways
> to autooffset.reset="largest" but it's still consuming items previously in
> the queue, and I'm not sure if the problem is on my end.
>
> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the
> latest offset for the queue itself (not the consumer)?
>
> Elben
>
>
> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > John,
> >
> > You can achieve that via the autooffset.reset="largest" config option
> > in the consumer.
> >
> > Thanks,
> > Neha
> >
> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com> wrote:
> > > Hi Neha:
> > >
> > > Can you point me to the api to always get data from tail?
> > >
> > > Thanks
> > >
> > > -John
> > >
> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >wrote:
> > >
> > >> Elben,
> > >>
> > >> Or sensei can provide a restart from tail API, that will delete the
> > offset
> > >> state from ZK and restart its Kafka consumer. Currently that is the
> best
> > >> way to have a Kafka consumer restart fetching latest data.
> > >>
> > >> Thanks,
> > >> Neha
> > >> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
> > >>
> > >> > I think in the newer kafka release, there is a tool to update the
> > version
> > >> > for a client.
> > >> >
> > >> > Jun, can you elaborate on that?
> > >> >
> > >> > Thanks
> > >> >
> > >> > -John
> > >> >
> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > We're using use kafka+sensei at our company and we'd love to be
> > able to
> > >> > > start at the end of the queue (latest entries). The use case is
> when
> > >> > sensei
> > >> > > goes down or has to be restarted, we only care about indexing the
> > >> latest
> > >> > > items.
> > >> > >
> > >> > > A workaround I guess is to stop sensei, manually change the high
> > >> > watermark
> > >> > > in kafka, then restart sensei.
> > >> > >
> > >> > > Elben
> > >> > >
> > >> > >
> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com>
> wrote:
> > >> > >
> > >> > > > Casey,
> > >> > > >
> > >> > > > You can't get offsets using zk-based consumer now. We are still
> > >> trying
> > >> > to
> > >> > > > figure about how to support that in the future.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
> > >> > > >
> > >> > > > > Hello,
> > >> > > > >
> > >> > > > > Just out of curiosity, how would you get the offsets when
> using
> > a
> > >> > > > zk-based
> > >> > > > > consumer? I don't see anything that seems to let you do this
> > >> easily.
> > >> > > > >
> > >> > > > > Casey
> > >> > > > >
> > >> > > > > ________________________________________
> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> > >> > > > > To: kafka-users@incubator.apache.org
> > >> > > > > Subject: Re: Replay
> > >> > > > >
> > >> > > > > With simple consumer, you can start from any valid offset that
> > you
> > >> > keep
> > >> > > > > track off.
> > >> > > > >
> > >> > > > > With the high level (zk-based) consumer: you will need to stop
> > all
> > >> > > > > consumers, and then update the offsets in zookeeper (see the
> > >> > > > > ImportZkOffsets tool in the tools package) with the offsets
> you
> > >> want
> > >> > to
> > >> > > > > replay from before restarting your consumer.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > >
> > >> > > > > Joel
> > >> > > > >
> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
> > JainK@bit-sys.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
> > >> > messages
> > >> > > > > from
> > >> > > > > > an "offset" that may have been committed before?
> > >> > > > > >
> > >> > > > > > Thanks for the help,
> > >> > > > > > Kokil Jain
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: Replay
Posted by Elben Shira <el...@gmail.com>.
Hi Neha,
What does it mean to fetch the latest data? If there are items in the kafka
queue and no new items are coming in, would autooffset.reset="largest" wait
for new items or would it consume some of the existing items in the queue?
I'm asking this because I've modified the sensei gateways
to autooffset.reset="largest" but it's still consuming items previously in
the queue, and I'm not sure if the problem is on my end.
I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the
latest offset for the queue itself (not the consumer)?
Elben
On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <ne...@gmail.com>wrote:
> John,
>
> You can achieve that via the autooffset.reset="largest" config option
> in the consumer.
>
> Thanks,
> Neha
>
> On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com> wrote:
> > Hi Neha:
> >
> > Can you point me to the api to always get data from tail?
> >
> > Thanks
> >
> > -John
> >
> > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Elben,
> >>
> >> Or sensei can provide a restart from tail API, that will delete the
> offset
> >> state from ZK and restart its Kafka consumer. Currently that is the best
> >> way to have a Kafka consumer restart fetching latest data.
> >>
> >> Thanks,
> >> Neha
> >> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
> >>
> >> > I think in the newer kafka release, there is a tool to update the
> version
> >> > for a client.
> >> >
> >> > Jun, can you elaborate on that?
> >> >
> >> > Thanks
> >> >
> >> > -John
> >> >
> >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
> >> wrote:
> >> >
> >> > > We're using use kafka+sensei at our company and we'd love to be
> able to
> >> > > start at the end of the queue (latest entries). The use case is when
> >> > sensei
> >> > > goes down or has to be restarted, we only care about indexing the
> >> latest
> >> > > items.
> >> > >
> >> > > A workaround I guess is to stop sensei, manually change the high
> >> > watermark
> >> > > in kafka, then restart sensei.
> >> > >
> >> > > Elben
> >> > >
> >> > >
> >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Casey,
> >> > > >
> >> > > > You can't get offsets using zk-based consumer now. We are still
> >> trying
> >> > to
> >> > > > figure about how to support that in the future.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> >> > > > Casey.Sybrandy@six3systems.com> wrote:
> >> > > >
> >> > > > > Hello,
> >> > > > >
> >> > > > > Just out of curiosity, how would you get the offsets when using
> a
> >> > > > zk-based
> >> > > > > consumer? I don't see anything that seems to let you do this
> >> easily.
> >> > > > >
> >> > > > > Casey
> >> > > > >
> >> > > > > ________________________________________
> >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> >> > > > > To: kafka-users@incubator.apache.org
> >> > > > > Subject: Re: Replay
> >> > > > >
> >> > > > > With simple consumer, you can start from any valid offset that
> you
> >> > keep
> >> > > > > track off.
> >> > > > >
> >> > > > > With the high level (zk-based) consumer: you will need to stop
> all
> >> > > > > consumers, and then update the offsets in zookeeper (see the
> >> > > > > ImportZkOffsets tool in the tools package) with the offsets you
> >> want
> >> > to
> >> > > > > replay from before restarting your consumer.
> >> > > > >
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Joel
> >> > > > >
> >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
> JainK@bit-sys.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
> >> > messages
> >> > > > > from
> >> > > > > > an "offset" that may have been committed before?
> >> > > > > >
> >> > > > > > Thanks for the help,
> >> > > > > > Kokil Jain
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: Replay
Posted by John Wang <jo...@gmail.com>.
Thanks Neha!
-John
On Thu, Mar 22, 2012 at 7:05 PM, Neha Narkhede <ne...@gmail.com>wrote:
> John,
>
> You can achieve that via the autooffset.reset="largest" config option
> in the consumer.
>
> Thanks,
> Neha
>
> On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com> wrote:
> > Hi Neha:
> >
> > Can you point me to the api to always get data from tail?
> >
> > Thanks
> >
> > -John
> >
> > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Elben,
> >>
> >> Or sensei can provide a restart from tail API, that will delete the
> offset
> >> state from ZK and restart its Kafka consumer. Currently that is the best
> >> way to have a Kafka consumer restart fetching latest data.
> >>
> >> Thanks,
> >> Neha
> >> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
> >>
> >> > I think in the newer kafka release, there is a tool to update the
> version
> >> > for a client.
> >> >
> >> > Jun, can you elaborate on that?
> >> >
> >> > Thanks
> >> >
> >> > -John
> >> >
> >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
> >> wrote:
> >> >
> >> > > We're using use kafka+sensei at our company and we'd love to be
> able to
> >> > > start at the end of the queue (latest entries). The use case is when
> >> > sensei
> >> > > goes down or has to be restarted, we only care about indexing the
> >> latest
> >> > > items.
> >> > >
> >> > > A workaround I guess is to stop sensei, manually change the high
> >> > watermark
> >> > > in kafka, then restart sensei.
> >> > >
> >> > > Elben
> >> > >
> >> > >
> >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Casey,
> >> > > >
> >> > > > You can't get offsets using zk-based consumer now. We are still
> >> trying
> >> > to
> >> > > > figure about how to support that in the future.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> >> > > > Casey.Sybrandy@six3systems.com> wrote:
> >> > > >
> >> > > > > Hello,
> >> > > > >
> >> > > > > Just out of curiosity, how would you get the offsets when using
> a
> >> > > > zk-based
> >> > > > > consumer? I don't see anything that seems to let you do this
> >> easily.
> >> > > > >
> >> > > > > Casey
> >> > > > >
> >> > > > > ________________________________________
> >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> >> > > > > To: kafka-users@incubator.apache.org
> >> > > > > Subject: Re: Replay
> >> > > > >
> >> > > > > With simple consumer, you can start from any valid offset that
> you
> >> > keep
> >> > > > > track off.
> >> > > > >
> >> > > > > With the high level (zk-based) consumer: you will need to stop
> all
> >> > > > > consumers, and then update the offsets in zookeeper (see the
> >> > > > > ImportZkOffsets tool in the tools package) with the offsets you
> >> want
> >> > to
> >> > > > > replay from before restarting your consumer.
> >> > > > >
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Joel
> >> > > > >
> >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
> JainK@bit-sys.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
> >> > messages
> >> > > > > from
> >> > > > > > an "offset" that may have been committed before?
> >> > > > > >
> >> > > > > > Thanks for the help,
> >> > > > > > Kokil Jain
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: Replay
Posted by Neha Narkhede <ne...@gmail.com>.
John,
You can achieve that via the autooffset.reset="largest" config option
in the consumer.
Thanks,
Neha
On Thu, Mar 22, 2012 at 6:15 PM, John Wang <jo...@gmail.com> wrote:
> Hi Neha:
>
> Can you point me to the api to always get data from tail?
>
> Thanks
>
> -John
>
> On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Elben,
>>
>> Or sensei can provide a restart from tail API, that will delete the offset
>> state from ZK and restart its Kafka consumer. Currently that is the best
>> way to have a Kafka consumer restart fetching latest data.
>>
>> Thanks,
>> Neha
>> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
>>
>> > I think in the newer kafka release, there is a tool to update the version
>> > for a client.
>> >
>> > Jun, can you elaborate on that?
>> >
>> > Thanks
>> >
>> > -John
>> >
>> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
>> wrote:
>> >
>> > > We're using use kafka+sensei at our company and we'd love to be able to
>> > > start at the end of the queue (latest entries). The use case is when
>> > sensei
>> > > goes down or has to be restarted, we only care about indexing the
>> latest
>> > > items.
>> > >
>> > > A workaround I guess is to stop sensei, manually change the high
>> > watermark
>> > > in kafka, then restart sensei.
>> > >
>> > > Elben
>> > >
>> > >
>> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
>> > >
>> > > > Casey,
>> > > >
>> > > > You can't get offsets using zk-based consumer now. We are still
>> trying
>> > to
>> > > > figure about how to support that in the future.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
>> > > > Casey.Sybrandy@six3systems.com> wrote:
>> > > >
>> > > > > Hello,
>> > > > >
>> > > > > Just out of curiosity, how would you get the offsets when using a
>> > > > zk-based
>> > > > > consumer? I don't see anything that seems to let you do this
>> easily.
>> > > > >
>> > > > > Casey
>> > > > >
>> > > > > ________________________________________
>> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
>> > > > > Sent: Monday, March 12, 2012 5:51 PM
>> > > > > To: kafka-users@incubator.apache.org
>> > > > > Subject: Re: Replay
>> > > > >
>> > > > > With simple consumer, you can start from any valid offset that you
>> > keep
>> > > > > track off.
>> > > > >
>> > > > > With the high level (zk-based) consumer: you will need to stop all
>> > > > > consumers, and then update the offsets in zookeeper (see the
>> > > > > ImportZkOffsets tool in the tools package) with the offsets you
>> want
>> > to
>> > > > > replay from before restarting your consumer.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Joel
>> > > > >
>> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
>> > messages
>> > > > > from
>> > > > > > an "offset" that may have been committed before?
>> > > > > >
>> > > > > > Thanks for the help,
>> > > > > > Kokil Jain
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
Re: Replay
Posted by John Wang <jo...@gmail.com>.
Hi Neha:
Can you point me to the api to always get data from tail?
Thanks
-John
On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Elben,
>
> Or sensei can provide a restart from tail API, that will delete the offset
> state from ZK and restart its Kafka consumer. Currently that is the best
> way to have a Kafka consumer restart fetching latest data.
>
> Thanks,
> Neha
> On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
>
> > I think in the newer kafka release, there is a tool to update the version
> > for a client.
> >
> > Jun, can you elaborate on that?
> >
> > Thanks
> >
> > -John
> >
> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com>
> wrote:
> >
> > > We're using use kafka+sensei at our company and we'd love to be able to
> > > start at the end of the queue (latest entries). The use case is when
> > sensei
> > > goes down or has to be restarted, we only care about indexing the
> latest
> > > items.
> > >
> > > A workaround I guess is to stop sensei, manually change the high
> > watermark
> > > in kafka, then restart sensei.
> > >
> > > Elben
> > >
> > >
> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Casey,
> > > >
> > > > You can't get offsets using zk-based consumer now. We are still
> trying
> > to
> > > > figure about how to support that in the future.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> > > > Casey.Sybrandy@six3systems.com> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Just out of curiosity, how would you get the offsets when using a
> > > > zk-based
> > > > > consumer? I don't see anything that seems to let you do this
> easily.
> > > > >
> > > > > Casey
> > > > >
> > > > > ________________________________________
> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> > > > > Sent: Monday, March 12, 2012 5:51 PM
> > > > > To: kafka-users@incubator.apache.org
> > > > > Subject: Re: Replay
> > > > >
> > > > > With simple consumer, you can start from any valid offset that you
> > keep
> > > > > track off.
> > > > >
> > > > > With the high level (zk-based) consumer: you will need to stop all
> > > > > consumers, and then update the offsets in zookeeper (see the
> > > > > ImportZkOffsets tool in the tools package) with the offsets you
> want
> > to
> > > > > replay from before restarting your consumer.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Joel
> > > > >
> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
> > messages
> > > > > from
> > > > > > an "offset" that may have been committed before?
> > > > > >
> > > > > > Thanks for the help,
> > > > > > Kokil Jain
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Replay
Posted by Neha Narkhede <ne...@gmail.com>.
Elben,
Or sensei can provide a restart from tail API, that will delete the offset
state from ZK and restart its Kafka consumer. Currently that is the best
way to have a Kafka consumer restart fetching latest data.
Thanks,
Neha
On Mar 22, 2012 4:39 PM, "John Wang" <jo...@gmail.com> wrote:
> I think in the newer kafka release, there is a tool to update the version
> for a client.
>
> Jun, can you elaborate on that?
>
> Thanks
>
> -John
>
> On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com> wrote:
>
> > We're using use kafka+sensei at our company and we'd love to be able to
> > start at the end of the queue (latest entries). The use case is when
> sensei
> > goes down or has to be restarted, we only care about indexing the latest
> > items.
> >
> > A workaround I guess is to stop sensei, manually change the high
> watermark
> > in kafka, then restart sensei.
> >
> > Elben
> >
> >
> > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Casey,
> > >
> > > You can't get offsets using zk-based consumer now. We are still trying
> to
> > > figure about how to support that in the future.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> > > Casey.Sybrandy@six3systems.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > Just out of curiosity, how would you get the offsets when using a
> > > zk-based
> > > > consumer? I don't see anything that seems to let you do this easily.
> > > >
> > > > Casey
> > > >
> > > > ________________________________________
> > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> > > > Sent: Monday, March 12, 2012 5:51 PM
> > > > To: kafka-users@incubator.apache.org
> > > > Subject: Re: Replay
> > > >
> > > > With simple consumer, you can start from any valid offset that you
> keep
> > > > track off.
> > > >
> > > > With the high level (zk-based) consumer: you will need to stop all
> > > > consumers, and then update the offsets in zookeeper (see the
> > > > ImportZkOffsets tool in the tools package) with the offsets you want
> to
> > > > replay from before restarting your consumer.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
> messages
> > > > from
> > > > > an "offset" that may have been committed before?
> > > > >
> > > > > Thanks for the help,
> > > > > Kokil Jain
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Replay
Posted by John Wang <jo...@gmail.com>.
I think in the newer kafka release, there is a tool to update the version
for a client.
Jun, can you elaborate on that?
Thanks
-John
On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <el...@gmail.com> wrote:
> We're using use kafka+sensei at our company and we'd love to be able to
> start at the end of the queue (latest entries). The use case is when sensei
> goes down or has to be restarted, we only care about indexing the latest
> items.
>
> A workaround I guess is to stop sensei, manually change the high watermark
> in kafka, then restart sensei.
>
> Elben
>
>
> On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Casey,
> >
> > You can't get offsets using zk-based consumer now. We are still trying to
> > figure about how to support that in the future.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> > Casey.Sybrandy@six3systems.com> wrote:
> >
> > > Hello,
> > >
> > > Just out of curiosity, how would you get the offsets when using a
> > zk-based
> > > consumer? I don't see anything that seems to let you do this easily.
> > >
> > > Casey
> > >
> > > ________________________________________
> > > From: Joel Koshy [jjkoshy.w@gmail.com]
> > > Sent: Monday, March 12, 2012 5:51 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Replay
> > >
> > > With simple consumer, you can start from any valid offset that you keep
> > > track off.
> > >
> > > With the high level (zk-based) consumer: you will need to stop all
> > > consumers, and then update the offsets in zookeeper (see the
> > > ImportZkOffsets tool in the tools package) with the offsets you want to
> > > replay from before restarting your consumer.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry, new to Kafka. Is it possible for a consumer to replay messages
> > > from
> > > > an "offset" that may have been committed before?
> > > >
> > > > Thanks for the help,
> > > > Kokil Jain
> > > >
> > > >
> > >
> >
>
Re: Replay
Posted by Elben Shira <el...@gmail.com>.
We're using use kafka+sensei at our company and we'd love to be able to
start at the end of the queue (latest entries). The use case is when sensei
goes down or has to be restarted, we only care about indexing the latest
items.
A workaround I guess is to stop sensei, manually change the high watermark
in kafka, then restart sensei.
Elben
On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <ju...@gmail.com> wrote:
> Casey,
>
> You can't get offsets using zk-based consumer now. We are still trying to
> figure about how to support that in the future.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > Just out of curiosity, how would you get the offsets when using a
> zk-based
> > consumer? I don't see anything that seems to let you do this easily.
> >
> > Casey
> >
> > ________________________________________
> > From: Joel Koshy [jjkoshy.w@gmail.com]
> > Sent: Monday, March 12, 2012 5:51 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Replay
> >
> > With simple consumer, you can start from any valid offset that you keep
> > track off.
> >
> > With the high level (zk-based) consumer: you will need to stop all
> > consumers, and then update the offsets in zookeeper (see the
> > ImportZkOffsets tool in the tools package) with the offsets you want to
> > replay from before restarting your consumer.
> >
> > Thanks,
> >
> > Joel
> >
> > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com> wrote:
> >
> > > Hi,
> > >
> > > Sorry, new to Kafka. Is it possible for a consumer to replay messages
> > from
> > > an "offset" that may have been committed before?
> > >
> > > Thanks for the help,
> > > Kokil Jain
> > >
> > >
> >
>
Re: Replay
Posted by Jun Rao <ju...@gmail.com>.
Casey,
You can't get offsets using zk-based consumer now. We are still trying to
figure about how to support that in the future.
Thanks,
Jun
On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> Just out of curiosity, how would you get the offsets when using a zk-based
> consumer? I don't see anything that seems to let you do this easily.
>
> Casey
>
> ________________________________________
> From: Joel Koshy [jjkoshy.w@gmail.com]
> Sent: Monday, March 12, 2012 5:51 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Replay
>
> With simple consumer, you can start from any valid offset that you keep
> track off.
>
> With the high level (zk-based) consumer: you will need to stop all
> consumers, and then update the offsets in zookeeper (see the
> ImportZkOffsets tool in the tools package) with the offsets you want to
> replay from before restarting your consumer.
>
> Thanks,
>
> Joel
>
> On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com> wrote:
>
> > Hi,
> >
> > Sorry, new to Kafka. Is it possible for a consumer to replay messages
> from
> > an "offset" that may have been committed before?
> >
> > Thanks for the help,
> > Kokil Jain
> >
> >
>
RE: Replay
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Hello,
Just out of curiosity, how would you get the offsets when using a zk-based consumer? I don't see anything that seems to let you do this easily.
Casey
________________________________________
From: Joel Koshy [jjkoshy.w@gmail.com]
Sent: Monday, March 12, 2012 5:51 PM
To: kafka-users@incubator.apache.org
Subject: Re: Replay
With simple consumer, you can start from any valid offset that you keep
track off.
With the high level (zk-based) consumer: you will need to stop all
consumers, and then update the offsets in zookeeper (see the
ImportZkOffsets tool in the tools package) with the offsets you want to
replay from before restarting your consumer.
Thanks,
Joel
On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com> wrote:
> Hi,
>
> Sorry, new to Kafka. Is it possible for a consumer to replay messages from
> an "offset" that may have been committed before?
>
> Thanks for the help,
> Kokil Jain
>
>
Re: Replay
Posted by Joel Koshy <jj...@gmail.com>.
With simple consumer, you can start from any valid offset that you keep
track off.
With the high level (zk-based) consumer: you will need to stop all
consumers, and then update the offsets in zookeeper (see the
ImportZkOffsets tool in the tools package) with the offsets you want to
replay from before restarting your consumer.
Thanks,
Joel
On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <Ja...@bit-sys.com> wrote:
> Hi,
>
> Sorry, new to Kafka. Is it possible for a consumer to replay messages from
> an "offset" that may have been committed before?
>
> Thanks for the help,
> Kokil Jain
>
>