You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Akhilesh Pathodia <pa...@gmail.com> on 2015/12/18 14:18:57 UTC
how to reset kafka offset in zookeeper
Hi,
I want to reset the kafka offset in zookeeper so that the consumer will
start reading messages from first offset. I am using flume as a consumer to
kafka. I have set the kafka property kafka.auto.offset.reset to "smallest",
but it does not reset the offset in zookeeper and that's why flume will not
read messages from first offset.
Is there any way to reset kafka offset in zookeeper?
Thanks,
Akhilesh
Re: how to reset kafka offset in zookeeper
Posted by Todd Palino <tp...@gmail.com>.
The way to reset to smallest is to stop the consumer, delete the consumer
group from Zookeeper, and then restart with the property set to smallest.
Once your consumer has recreated the group and committed offsets, you can
change the auto.offset.reset property back to largest (if that is your
preference).
-Todd
On Friday, December 18, 2015, Akhilesh Pathodia <pa...@gmail.com>
wrote:
> Hi,
>
> I want to reset the kafka offset in zookeeper so that the consumer will
> start reading messages from first offset. I am using flume as a consumer to
> kafka. I have set the kafka property kafka.auto.offset.reset to "smallest",
> but it does not reset the offset in zookeeper and that's why flume will not
> read messages from first offset.
>
> Is there any way to reset kafka offset in zookeeper?
>
> Thanks,
> Akhilesh
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by James Cheng <jc...@tivo.com>.
This page describes what Kafka stores in Zookeeper:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
It looks like the info for a particular consumer groupId is stored at:
/consumers/<groupId>/
According to https://community.cloudera.com/t5/Cloudera-Labs/Kafka-Parcels/td-p/20392, Cloudera's default config is to put a zookeeper chroot of "/kafka"
So you might have to look at
/kafka/consumers/<groupId>
-James
> On Dec 19, 2015, at 3:14 PM, Todd Palino <tp...@gmail.com> wrote:
>
> There’s no simple command. You’ll need to use either zookeeper-shell.sh or
> zkCli.sh or something similar that lets you explore and edit Zookeeper and
> do a recursive delete on the group name in the consumers tree. I’m not sure
> how Cloudera’s interface differs, however, or if they provide a separate
> tool for deleting a consumer group.
>
> -Todd
>
>
> On Sat, Dec 19, 2015 at 11:34 AM, Akhilesh Pathodia <
> pathodia.akhilesh@gmail.com> wrote:
>
>> What is the command to delete group from zookeeper? I dont find
>> /consumer/ directory? I am using cloudera, is there any place on cloudera
>> manager where I can delete the group?
>>
>> Thanks
>>
>> On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino <tp...@gmail.com> wrote:
>>
>>> If what you want to do is reset to smallest, all you need to do is stop
>> the
>>> consumer, delete the group from Zookeeper, and restart the consumer. It
>>> will automatically create the group again.
>>>
>>> You only need to export the offsets first if you later need to reset to
>>> where you were in the partitions.
>>>
>>> -Todd
>>>
>>> On Saturday, December 19, 2015, Akhilesh Pathodia <
>>> pathodia.akhilesh@gmail.com> wrote:
>>>
>>>> What is the process for deleting the consumer group from zookeeper?
>>> Should
>>>> I export offset, delete and then import?
>>>>
>>>> Thanks,
>>>> Akhilesh
>>>>
>>>> On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tpalino@gmail.com
>>>> <javascript:;>> wrote:
>>>>
>>>>> Yes, that’s right. It’s just work for no real gain :)
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <
>>> marko.bonaci@sematext.com
>>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> Hmm, I guess you're right Tod :)
>>>>>> Just to confirm, you meant that, while you're changing the exported
>>>> file
>>>>> it
>>>>>> might happen that one of the segment files becomes eligible for
>>> cleanup
>>>>> by
>>>>>> retention, which would then make the imported offsets out of range?
>>>>>>
>>>>>> Marko Bonaći
>>>>>> Monitoring | Alerting | Anomaly Detection | Centralized Log
>>> Management
>>>>>> Solr & Elasticsearch Support
>>>>>> Sematext <http://sematext.com/> | Contact
>>>>>> <http://sematext.com/about/contact.html>
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com
>>>> <javascript:;>> wrote:
>>>>>>
>>>>>>> That works if you want to set to an arbitrary offset, Marko.
>>> However
>>>> in
>>>>>> the
>>>>>>> case the OP described, wanting to reset to smallest, it is better
>>> to
>>>>> just
>>>>>>> delete the consumer group and start the consumer with
>>>> auto.offset.reset
>>>>>> set
>>>>>>> to smallest. The reason is that while you can pull the current
>>>> smallest
>>>>>>> offsets from the brokers and set them in Zookeeper for the
>>> consumer,
>>>> by
>>>>>> the
>>>>>>> time you do that the smallest offset is likely no longer valid.
>>> This
>>>>>> means
>>>>>>> you’re going to resort to the offset reset logic anyways.
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
>>>>> marko.bonaci@sematext.com <javascript:;>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You can also do this:
>>>>>>>> 1. stop consumers
>>>>>>>> 2. export offsets from ZK
>>>>>>>> 3. make changes to the exported file
>>>>>>>> 4. import offsets to ZK
>>>>>>>> 5. start consumers
>>>>>>>>
>>>>>>>> e.g.
>>>>>>>> bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group
>>>> group-name
>>>>>>>> --output-file /tmp/zk-offsets --zkconnect localhost:2181
>>>>>>>> bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
>>>>>>>> /tmp/zk-offsets --zkconnect localhost:2181
>>>>>>>>
>>>>>>>> Marko Bonaći
>>>>>>>> Monitoring | Alerting | Anomaly Detection | Centralized Log
>>>>> Management
>>>>>>>> Solr & Elasticsearch Support
>>>>>>>> Sematext <http://sematext.com/> | Contact
>>>>>>>> <http://sematext.com/about/contact.html>
>>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <
>>> jens.rantil@tink.se
>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I noticed that a consumer in the new consumer API supports
>>>> setting
>>>>>> the
>>>>>>>>> offset for a partition to beginning. I assume doing so also
>>> would
>>>>>>> update
>>>>>>>>> the offset in Zookeeper eventually.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Jens
>>>>>>>>>
>>>>>>>>> On Friday, December 18, 2015, Akhilesh Pathodia <
>>>>>>>>> pathodia.akhilesh@gmail.com <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I want to reset the kafka offset in zookeeper so that the
>>>>> consumer
>>>>>>> will
>>>>>>>>>> start reading messages from first offset. I am using flume
>>> as a
>>>>>>>> consumer
>>>>>>>>> to
>>>>>>>>>> kafka. I have set the kafka property
>> kafka.auto.offset.reset
>>> to
>>>>>>>>> "smallest",
>>>>>>>>>> but it does not reset the offset in zookeeper and that's
>> why
>>>>> flume
>>>>>>> will
>>>>>>>>> not
>>>>>>>>>> read messages from first offset.
>>>>>>>>>>
>>>>>>>>>> Is there any way to reset kafka offset in zookeeper?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Akhilesh
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jens Rantil
>>>>>>>>> Backend engineer
>>>>>>>>> Tink AB
>>>>>>>>>
>>>>>>>>> Email: jens.rantil@tink.se <javascript:;>
>>>>>>>>> Phone: +46 708 84 18 32
>>>>>>>>> Web: www.tink.se
>>>>>>>>>
>>>>>>>>> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
>>>>>>>>> <
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
>>>>>>>>>>
>>>>>>>>> Twitter <https://twitter.com/tink>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *—-*
>>>>>>> *Todd Palino*
>>>>>>> Staff Site Reliability Engineer
>>>>>>> Data Infrastructure Streaming
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> linkedin.com/in/toddpalino
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *—-*
>>>>> *Todd Palino*
>>>>> Staff Site Reliability Engineer
>>>>> Data Infrastructure Streaming
>>>>>
>>>>>
>>>>>
>>>>> linkedin.com/in/toddpalino
>>>>>
>>>>
>>>
>>>
>>> --
>>> *—-*
>>> *Todd Palino*
>>> Staff Site Reliability Engineer
>>> Data Infrastructure Streaming
>>>
>>>
>>>
>>> linkedin.com/in/toddpalino
>>>
>>
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by Todd Palino <tp...@gmail.com>.
There’s no simple command. You’ll need to use either zookeeper-shell.sh or
zkCli.sh or something similar that lets you explore and edit Zookeeper and
do a recursive delete on the group name in the consumers tree. I’m not sure
how Cloudera’s interface differs, however, or if they provide a separate
tool for deleting a consumer group.
-Todd
On Sat, Dec 19, 2015 at 11:34 AM, Akhilesh Pathodia <
pathodia.akhilesh@gmail.com> wrote:
> What is the command to delete group from zookeeper? I dont find
> /consumer/ directory? I am using cloudera, is there any place on cloudera
> manager where I can delete the group?
>
> Thanks
>
> On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > If what you want to do is reset to smallest, all you need to do is stop
> the
> > consumer, delete the group from Zookeeper, and restart the consumer. It
> > will automatically create the group again.
> >
> > You only need to export the offsets first if you later need to reset to
> > where you were in the partitions.
> >
> > -Todd
> >
> > On Saturday, December 19, 2015, Akhilesh Pathodia <
> > pathodia.akhilesh@gmail.com> wrote:
> >
> > > What is the process for deleting the consumer group from zookeeper?
> > Should
> > > I export offset, delete and then import?
> > >
> > > Thanks,
> > > Akhilesh
> > >
> > > On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tpalino@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > Yes, that’s right. It’s just work for no real gain :)
> > > >
> > > > -Todd
> > > >
> > > > On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <
> > marko.bonaci@sematext.com
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > Hmm, I guess you're right Tod :)
> > > > > Just to confirm, you meant that, while you're changing the exported
> > > file
> > > > it
> > > > > might happen that one of the segment files becomes eligible for
> > cleanup
> > > > by
> > > > > retention, which would then make the imported offsets out of range?
> > > > >
> > > > > Marko Bonaći
> > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support
> > > > > Sematext <http://sematext.com/> | Contact
> > > > > <http://sematext.com/about/contact.html>
> > > > >
> > > > > On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com
> > > <javascript:;>> wrote:
> > > > >
> > > > > > That works if you want to set to an arbitrary offset, Marko.
> > However
> > > in
> > > > > the
> > > > > > case the OP described, wanting to reset to smallest, it is better
> > to
> > > > just
> > > > > > delete the consumer group and start the consumer with
> > > auto.offset.reset
> > > > > set
> > > > > > to smallest. The reason is that while you can pull the current
> > > smallest
> > > > > > offsets from the brokers and set them in Zookeeper for the
> > consumer,
> > > by
> > > > > the
> > > > > > time you do that the smallest offset is likely no longer valid.
> > This
> > > > > means
> > > > > > you’re going to resort to the offset reset logic anyways.
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > >
> > > > > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
> > > > marko.bonaci@sematext.com <javascript:;>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > You can also do this:
> > > > > > > 1. stop consumers
> > > > > > > 2. export offsets from ZK
> > > > > > > 3. make changes to the exported file
> > > > > > > 4. import offsets to ZK
> > > > > > > 5. start consumers
> > > > > > >
> > > > > > > e.g.
> > > > > > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group
> > > group-name
> > > > > > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > > > > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > > > > > /tmp/zk-offsets --zkconnect localhost:2181
> > > > > > >
> > > > > > > Marko Bonaći
> > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support
> > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > <http://sematext.com/about/contact.html>
> > > > > > >
> > > > > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <
> > jens.rantil@tink.se
> > > <javascript:;>>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I noticed that a consumer in the new consumer API supports
> > > setting
> > > > > the
> > > > > > > > offset for a partition to beginning. I assume doing so also
> > would
> > > > > > update
> > > > > > > > the offset in Zookeeper eventually.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jens
> > > > > > > >
> > > > > > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > > > > > pathodia.akhilesh@gmail.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I want to reset the kafka offset in zookeeper so that the
> > > > consumer
> > > > > > will
> > > > > > > > > start reading messages from first offset. I am using flume
> > as a
> > > > > > > consumer
> > > > > > > > to
> > > > > > > > > kafka. I have set the kafka property
> kafka.auto.offset.reset
> > to
> > > > > > > > "smallest",
> > > > > > > > > but it does not reset the offset in zookeeper and that's
> why
> > > > flume
> > > > > > will
> > > > > > > > not
> > > > > > > > > read messages from first offset.
> > > > > > > > >
> > > > > > > > > Is there any way to reset kafka offset in zookeeper?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Akhilesh
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Jens Rantil
> > > > > > > > Backend engineer
> > > > > > > > Tink AB
> > > > > > > >
> > > > > > > > Email: jens.rantil@tink.se <javascript:;>
> > > > > > > > Phone: +46 708 84 18 32
> > > > > > > > Web: www.tink.se
> > > > > > > >
> > > > > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > > > > > >
> > > > > > > > Twitter <https://twitter.com/tink>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *—-*
> > > > > > *Todd Palino*
> > > > > > Staff Site Reliability Engineer
> > > > > > Data Infrastructure Streaming
> > > > > >
> > > > > >
> > > > > >
> > > > > > linkedin.com/in/toddpalino
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *—-*
> > > > *Todd Palino*
> > > > Staff Site Reliability Engineer
> > > > Data Infrastructure Streaming
> > > >
> > > >
> > > >
> > > > linkedin.com/in/toddpalino
> > > >
> > >
> >
> >
> > --
> > *—-*
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by Akhilesh Pathodia <pa...@gmail.com>.
What is the command to delete group from zookeeper? I dont find
/consumer/ directory? I am using cloudera, is there any place on cloudera
manager where I can delete the group?
Thanks
On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino <tp...@gmail.com> wrote:
> If what you want to do is reset to smallest, all you need to do is stop the
> consumer, delete the group from Zookeeper, and restart the consumer. It
> will automatically create the group again.
>
> You only need to export the offsets first if you later need to reset to
> where you were in the partitions.
>
> -Todd
>
> On Saturday, December 19, 2015, Akhilesh Pathodia <
> pathodia.akhilesh@gmail.com> wrote:
>
> > What is the process for deleting the consumer group from zookeeper?
> Should
> > I export offset, delete and then import?
> >
> > Thanks,
> > Akhilesh
> >
> > On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> >
> > > Yes, that’s right. It’s just work for no real gain :)
> > >
> > > -Todd
> > >
> > > On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <
> marko.bonaci@sematext.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > Hmm, I guess you're right Tod :)
> > > > Just to confirm, you meant that, while you're changing the exported
> > file
> > > it
> > > > might happen that one of the segment files becomes eligible for
> cleanup
> > > by
> > > > retention, which would then make the imported offsets out of range?
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > > That works if you want to set to an arbitrary offset, Marko.
> However
> > in
> > > > the
> > > > > case the OP described, wanting to reset to smallest, it is better
> to
> > > just
> > > > > delete the consumer group and start the consumer with
> > auto.offset.reset
> > > > set
> > > > > to smallest. The reason is that while you can pull the current
> > smallest
> > > > > offsets from the brokers and set them in Zookeeper for the
> consumer,
> > by
> > > > the
> > > > > time you do that the smallest offset is likely no longer valid.
> This
> > > > means
> > > > > you’re going to resort to the offset reset logic anyways.
> > > > >
> > > > > -Todd
> > > > >
> > > > >
> > > > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
> > > marko.bonaci@sematext.com <javascript:;>
> > > > >
> > > > > wrote:
> > > > >
> > > > > > You can also do this:
> > > > > > 1. stop consumers
> > > > > > 2. export offsets from ZK
> > > > > > 3. make changes to the exported file
> > > > > > 4. import offsets to ZK
> > > > > > 5. start consumers
> > > > > >
> > > > > > e.g.
> > > > > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group
> > group-name
> > > > > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > > > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > > > > /tmp/zk-offsets --zkconnect localhost:2181
> > > > > >
> > > > > > Marko Bonaći
> > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support
> > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > <http://sematext.com/about/contact.html>
> > > > > >
> > > > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <
> jens.rantil@tink.se
> > <javascript:;>>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I noticed that a consumer in the new consumer API supports
> > setting
> > > > the
> > > > > > > offset for a partition to beginning. I assume doing so also
> would
> > > > > update
> > > > > > > the offset in Zookeeper eventually.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jens
> > > > > > >
> > > > > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > > > > pathodia.akhilesh@gmail.com <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I want to reset the kafka offset in zookeeper so that the
> > > consumer
> > > > > will
> > > > > > > > start reading messages from first offset. I am using flume
> as a
> > > > > > consumer
> > > > > > > to
> > > > > > > > kafka. I have set the kafka property kafka.auto.offset.reset
> to
> > > > > > > "smallest",
> > > > > > > > but it does not reset the offset in zookeeper and that's why
> > > flume
> > > > > will
> > > > > > > not
> > > > > > > > read messages from first offset.
> > > > > > > >
> > > > > > > > Is there any way to reset kafka offset in zookeeper?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Akhilesh
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Jens Rantil
> > > > > > > Backend engineer
> > > > > > > Tink AB
> > > > > > >
> > > > > > > Email: jens.rantil@tink.se <javascript:;>
> > > > > > > Phone: +46 708 84 18 32
> > > > > > > Web: www.tink.se
> > > > > > >
> > > > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > > > > >
> > > > > > > Twitter <https://twitter.com/tink>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *—-*
> > > > > *Todd Palino*
> > > > > Staff Site Reliability Engineer
> > > > > Data Infrastructure Streaming
> > > > >
> > > > >
> > > > >
> > > > > linkedin.com/in/toddpalino
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *—-*
> > > *Todd Palino*
> > > Staff Site Reliability Engineer
> > > Data Infrastructure Streaming
> > >
> > >
> > >
> > > linkedin.com/in/toddpalino
> > >
> >
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>
Re: how to reset kafka offset in zookeeper
Posted by Todd Palino <tp...@gmail.com>.
If what you want to do is reset to smallest, all you need to do is stop the
consumer, delete the group from Zookeeper, and restart the consumer. It
will automatically create the group again.
You only need to export the offsets first if you later need to reset to
where you were in the partitions.
-Todd
On Saturday, December 19, 2015, Akhilesh Pathodia <
pathodia.akhilesh@gmail.com> wrote:
> What is the process for deleting the consumer group from zookeeper? Should
> I export offset, delete and then import?
>
> Thanks,
> Akhilesh
>
> On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tpalino@gmail.com
> <javascript:;>> wrote:
>
> > Yes, that’s right. It’s just work for no real gain :)
> >
> > -Todd
> >
> > On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <marko.bonaci@sematext.com
> <javascript:;>>
> > wrote:
> >
> > > Hmm, I guess you're right Tod :)
> > > Just to confirm, you meant that, while you're changing the exported
> file
> > it
> > > might happen that one of the segment files becomes eligible for cleanup
> > by
> > > retention, which would then make the imported offsets out of range?
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com
> <javascript:;>> wrote:
> > >
> > > > That works if you want to set to an arbitrary offset, Marko. However
> in
> > > the
> > > > case the OP described, wanting to reset to smallest, it is better to
> > just
> > > > delete the consumer group and start the consumer with
> auto.offset.reset
> > > set
> > > > to smallest. The reason is that while you can pull the current
> smallest
> > > > offsets from the brokers and set them in Zookeeper for the consumer,
> by
> > > the
> > > > time you do that the smallest offset is likely no longer valid. This
> > > means
> > > > you’re going to resort to the offset reset logic anyways.
> > > >
> > > > -Todd
> > > >
> > > >
> > > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
> > marko.bonaci@sematext.com <javascript:;>
> > > >
> > > > wrote:
> > > >
> > > > > You can also do this:
> > > > > 1. stop consumers
> > > > > 2. export offsets from ZK
> > > > > 3. make changes to the exported file
> > > > > 4. import offsets to ZK
> > > > > 5. start consumers
> > > > >
> > > > > e.g.
> > > > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group
> group-name
> > > > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > > > /tmp/zk-offsets --zkconnect localhost:2181
> > > > >
> > > > > Marko Bonaći
> > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support
> > > > > Sematext <http://sematext.com/> | Contact
> > > > > <http://sematext.com/about/contact.html>
> > > > >
> > > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <jens.rantil@tink.se
> <javascript:;>>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I noticed that a consumer in the new consumer API supports
> setting
> > > the
> > > > > > offset for a partition to beginning. I assume doing so also would
> > > > update
> > > > > > the offset in Zookeeper eventually.
> > > > > >
> > > > > > Cheers,
> > > > > > Jens
> > > > > >
> > > > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > > > pathodia.akhilesh@gmail.com <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I want to reset the kafka offset in zookeeper so that the
> > consumer
> > > > will
> > > > > > > start reading messages from first offset. I am using flume as a
> > > > > consumer
> > > > > > to
> > > > > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > > > > "smallest",
> > > > > > > but it does not reset the offset in zookeeper and that's why
> > flume
> > > > will
> > > > > > not
> > > > > > > read messages from first offset.
> > > > > > >
> > > > > > > Is there any way to reset kafka offset in zookeeper?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Akhilesh
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jens Rantil
> > > > > > Backend engineer
> > > > > > Tink AB
> > > > > >
> > > > > > Email: jens.rantil@tink.se <javascript:;>
> > > > > > Phone: +46 708 84 18 32
> > > > > > Web: www.tink.se
> > > > > >
> > > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > > > >
> > > > > > Twitter <https://twitter.com/tink>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *—-*
> > > > *Todd Palino*
> > > > Staff Site Reliability Engineer
> > > > Data Infrastructure Streaming
> > > >
> > > >
> > > >
> > > > linkedin.com/in/toddpalino
> > > >
> > >
> >
> >
> >
> > --
> > *—-*
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by Akhilesh Pathodia <pa...@gmail.com>.
What is the process for deleting the consumer group from zookeeper? Should
I export offset, delete and then import?
Thanks,
Akhilesh
On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tp...@gmail.com> wrote:
> Yes, that’s right. It’s just work for no real gain :)
>
> -Todd
>
> On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <ma...@sematext.com>
> wrote:
>
> > Hmm, I guess you're right Tod :)
> > Just to confirm, you meant that, while you're changing the exported file
> it
> > might happen that one of the segment files becomes eligible for cleanup
> by
> > retention, which would then make the imported offsets out of range?
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > That works if you want to set to an arbitrary offset, Marko. However in
> > the
> > > case the OP described, wanting to reset to smallest, it is better to
> just
> > > delete the consumer group and start the consumer with auto.offset.reset
> > set
> > > to smallest. The reason is that while you can pull the current smallest
> > > offsets from the brokers and set them in Zookeeper for the consumer, by
> > the
> > > time you do that the smallest offset is likely no longer valid. This
> > means
> > > you’re going to resort to the offset reset logic anyways.
> > >
> > > -Todd
> > >
> > >
> > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
> marko.bonaci@sematext.com
> > >
> > > wrote:
> > >
> > > > You can also do this:
> > > > 1. stop consumers
> > > > 2. export offsets from ZK
> > > > 3. make changes to the exported file
> > > > 4. import offsets to ZK
> > > > 5. start consumers
> > > >
> > > > e.g.
> > > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> > > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > > /tmp/zk-offsets --zkconnect localhost:2181
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I noticed that a consumer in the new consumer API supports setting
> > the
> > > > > offset for a partition to beginning. I assume doing so also would
> > > update
> > > > > the offset in Zookeeper eventually.
> > > > >
> > > > > Cheers,
> > > > > Jens
> > > > >
> > > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > > pathodia.akhilesh@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I want to reset the kafka offset in zookeeper so that the
> consumer
> > > will
> > > > > > start reading messages from first offset. I am using flume as a
> > > > consumer
> > > > > to
> > > > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > > > "smallest",
> > > > > > but it does not reset the offset in zookeeper and that's why
> flume
> > > will
> > > > > not
> > > > > > read messages from first offset.
> > > > > >
> > > > > > Is there any way to reset kafka offset in zookeeper?
> > > > > >
> > > > > > Thanks,
> > > > > > Akhilesh
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Jens Rantil
> > > > > Backend engineer
> > > > > Tink AB
> > > > >
> > > > > Email: jens.rantil@tink.se
> > > > > Phone: +46 708 84 18 32
> > > > > Web: www.tink.se
> > > > >
> > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > > <
> > > > >
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > > >
> > > > > Twitter <https://twitter.com/tink>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *—-*
> > > *Todd Palino*
> > > Staff Site Reliability Engineer
> > > Data Infrastructure Streaming
> > >
> > >
> > >
> > > linkedin.com/in/toddpalino
> > >
> >
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>
Re: how to reset kafka offset in zookeeper
Posted by Todd Palino <tp...@gmail.com>.
Yes, that’s right. It’s just work for no real gain :)
-Todd
On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <ma...@sematext.com>
wrote:
> Hmm, I guess you're right Tod :)
> Just to confirm, you meant that, while you're changing the exported file it
> might happen that one of the segment files becomes eligible for cleanup by
> retention, which would then make the imported offsets out of range?
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > That works if you want to set to an arbitrary offset, Marko. However in
> the
> > case the OP described, wanting to reset to smallest, it is better to just
> > delete the consumer group and start the consumer with auto.offset.reset
> set
> > to smallest. The reason is that while you can pull the current smallest
> > offsets from the brokers and set them in Zookeeper for the consumer, by
> the
> > time you do that the smallest offset is likely no longer valid. This
> means
> > you’re going to resort to the offset reset logic anyways.
> >
> > -Todd
> >
> >
> > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <marko.bonaci@sematext.com
> >
> > wrote:
> >
> > > You can also do this:
> > > 1. stop consumers
> > > 2. export offsets from ZK
> > > 3. make changes to the exported file
> > > 4. import offsets to ZK
> > > 5. start consumers
> > >
> > > e.g.
> > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > /tmp/zk-offsets --zkconnect localhost:2181
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed that a consumer in the new consumer API supports setting
> the
> > > > offset for a partition to beginning. I assume doing so also would
> > update
> > > > the offset in Zookeeper eventually.
> > > >
> > > > Cheers,
> > > > Jens
> > > >
> > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > pathodia.akhilesh@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I want to reset the kafka offset in zookeeper so that the consumer
> > will
> > > > > start reading messages from first offset. I am using flume as a
> > > consumer
> > > > to
> > > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > > "smallest",
> > > > > but it does not reset the offset in zookeeper and that's why flume
> > will
> > > > not
> > > > > read messages from first offset.
> > > > >
> > > > > Is there any way to reset kafka offset in zookeeper?
> > > > >
> > > > > Thanks,
> > > > > Akhilesh
> > > > >
> > > >
> > > >
> > > > --
> > > > Jens Rantil
> > > > Backend engineer
> > > > Tink AB
> > > >
> > > > Email: jens.rantil@tink.se
> > > > Phone: +46 708 84 18 32
> > > > Web: www.tink.se
> > > >
> > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > <
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > >
> > > > Twitter <https://twitter.com/tink>
> > > >
> > >
> >
> >
> >
> > --
> > *—-*
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by Dana Powers <da...@gmail.com>.
If you don't like messing w/ ZK directly, another alternative is to
manually seek to offset 0 on all relevant topic-partitions (via
OffsetCommitRequest or your favorite client api) and change the
auto-offset-reset policy on your consumer to earliest/smallest. Bonus is
that this should also work for consumers that use kafka-backed offset
commit storage.
-Dana
On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <ma...@sematext.com>
wrote:
> Hmm, I guess you're right Tod :)
> Just to confirm, you meant that, while you're changing the exported file it
> might happen that one of the segment files becomes eligible for cleanup by
> retention, which would then make the imported offsets out of range?
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > That works if you want to set to an arbitrary offset, Marko. However in
> the
> > case the OP described, wanting to reset to smallest, it is better to just
> > delete the consumer group and start the consumer with auto.offset.reset
> set
> > to smallest. The reason is that while you can pull the current smallest
> > offsets from the brokers and set them in Zookeeper for the consumer, by
> the
> > time you do that the smallest offset is likely no longer valid. This
> means
> > you’re going to resort to the offset reset logic anyways.
> >
> > -Todd
> >
> >
> > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <marko.bonaci@sematext.com
> >
> > wrote:
> >
> > > You can also do this:
> > > 1. stop consumers
> > > 2. export offsets from ZK
> > > 3. make changes to the exported file
> > > 4. import offsets to ZK
> > > 5. start consumers
> > >
> > > e.g.
> > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > /tmp/zk-offsets --zkconnect localhost:2181
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed that a consumer in the new consumer API supports setting
> the
> > > > offset for a partition to beginning. I assume doing so also would
> > update
> > > > the offset in Zookeeper eventually.
> > > >
> > > > Cheers,
> > > > Jens
> > > >
> > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > pathodia.akhilesh@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I want to reset the kafka offset in zookeeper so that the consumer
> > will
> > > > > start reading messages from first offset. I am using flume as a
> > > consumer
> > > > to
> > > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > > "smallest",
> > > > > but it does not reset the offset in zookeeper and that's why flume
> > will
> > > > not
> > > > > read messages from first offset.
> > > > >
> > > > > Is there any way to reset kafka offset in zookeeper?
> > > > >
> > > > > Thanks,
> > > > > Akhilesh
> > > > >
> > > >
> > > >
> > > > --
> > > > Jens Rantil
> > > > Backend engineer
> > > > Tink AB
> > > >
> > > > Email: jens.rantil@tink.se
> > > > Phone: +46 708 84 18 32
> > > > Web: www.tink.se
> > > >
> > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > <
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > >
> > > > Twitter <https://twitter.com/tink>
> > > >
> > >
> >
> >
> >
> > --
> > *—-*
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>
Re: how to reset kafka offset in zookeeper
Posted by Marko Bonaći <ma...@sematext.com>.
Hmm, I guess you're right Tod :)
Just to confirm, you meant that, while you're changing the exported file it
might happen that one of the segment files becomes eligible for cleanup by
retention, which would then make the imported offsets out of range?
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tp...@gmail.com> wrote:
> That works if you want to set to an arbitrary offset, Marko. However in the
> case the OP described, wanting to reset to smallest, it is better to just
> delete the consumer group and start the consumer with auto.offset.reset set
> to smallest. The reason is that while you can pull the current smallest
> offsets from the brokers and set them in Zookeeper for the consumer, by the
> time you do that the smallest offset is likely no longer valid. This means
> you’re going to resort to the offset reset logic anyways.
>
> -Todd
>
>
> On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <ma...@sematext.com>
> wrote:
>
> > You can also do this:
> > 1. stop consumers
> > 2. export offsets from ZK
> > 3. make changes to the exported file
> > 4. import offsets to ZK
> > 5. start consumers
> >
> > e.g.
> > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > /tmp/zk-offsets --zkconnect localhost:2181
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se>
> wrote:
> >
> > > Hi,
> > >
> > > I noticed that a consumer in the new consumer API supports setting the
> > > offset for a partition to beginning. I assume doing so also would
> update
> > > the offset in Zookeeper eventually.
> > >
> > > Cheers,
> > > Jens
> > >
> > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > pathodia.akhilesh@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I want to reset the kafka offset in zookeeper so that the consumer
> will
> > > > start reading messages from first offset. I am using flume as a
> > consumer
> > > to
> > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > "smallest",
> > > > but it does not reset the offset in zookeeper and that's why flume
> will
> > > not
> > > > read messages from first offset.
> > > >
> > > > Is there any way to reset kafka offset in zookeeper?
> > > >
> > > > Thanks,
> > > > Akhilesh
> > > >
> > >
> > >
> > > --
> > > Jens Rantil
> > > Backend engineer
> > > Tink AB
> > >
> > > Email: jens.rantil@tink.se
> > > Phone: +46 708 84 18 32
> > > Web: www.tink.se
> > >
> > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > <
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > >
> > > Twitter <https://twitter.com/tink>
> > >
> >
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>
Re: how to reset kafka offset in zookeeper
Posted by Todd Palino <tp...@gmail.com>.
That works if you want to set to an arbitrary offset, Marko. However in the
case the OP described, wanting to reset to smallest, it is better to just
delete the consumer group and start the consumer with auto.offset.reset set
to smallest. The reason is that while you can pull the current smallest
offsets from the brokers and set them in Zookeeper for the consumer, by the
time you do that the smallest offset is likely no longer valid. This means
you’re going to resort to the offset reset logic anyways.
-Todd
On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <ma...@sematext.com>
wrote:
> You can also do this:
> 1. stop consumers
> 2. export offsets from ZK
> 3. make changes to the exported file
> 4. import offsets to ZK
> 5. start consumers
>
> e.g.
> bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> --output-file /tmp/zk-offsets --zkconnect localhost:2181
> bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> /tmp/zk-offsets --zkconnect localhost:2181
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se> wrote:
>
> > Hi,
> >
> > I noticed that a consumer in the new consumer API supports setting the
> > offset for a partition to beginning. I assume doing so also would update
> > the offset in Zookeeper eventually.
> >
> > Cheers,
> > Jens
> >
> > On Friday, December 18, 2015, Akhilesh Pathodia <
> > pathodia.akhilesh@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I want to reset the kafka offset in zookeeper so that the consumer will
> > > start reading messages from first offset. I am using flume as a
> consumer
> > to
> > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > "smallest",
> > > but it does not reset the offset in zookeeper and that's why flume will
> > not
> > > read messages from first offset.
> > >
> > > Is there any way to reset kafka offset in zookeeper?
> > >
> > > Thanks,
> > > Akhilesh
> > >
> >
> >
> > --
> > Jens Rantil
> > Backend engineer
> > Tink AB
> >
> > Email: jens.rantil@tink.se
> > Phone: +46 708 84 18 32
> > Web: www.tink.se
> >
> > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > <
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > >
> > Twitter <https://twitter.com/tink>
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: how to reset kafka offset in zookeeper
Posted by Marko Bonaći <ma...@sematext.com>.
You can also do this:
1. stop consumers
2. export offsets from ZK
3. make changes to the exported file
4. import offsets to ZK
5. start consumers
e.g.
bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
--output-file /tmp/zk-offsets --zkconnect localhost:2181
bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
/tmp/zk-offsets --zkconnect localhost:2181
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <je...@tink.se> wrote:
> Hi,
>
> I noticed that a consumer in the new consumer API supports setting the
> offset for a partition to beginning. I assume doing so also would update
> the offset in Zookeeper eventually.
>
> Cheers,
> Jens
>
> On Friday, December 18, 2015, Akhilesh Pathodia <
> pathodia.akhilesh@gmail.com>
> wrote:
>
> > Hi,
> >
> > I want to reset the kafka offset in zookeeper so that the consumer will
> > start reading messages from first offset. I am using flume as a consumer
> to
> > kafka. I have set the kafka property kafka.auto.offset.reset to
> "smallest",
> > but it does not reset the offset in zookeeper and that's why flume will
> not
> > read messages from first offset.
> >
> > Is there any way to reset kafka offset in zookeeper?
> >
> > Thanks,
> > Akhilesh
> >
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> >
> Twitter <https://twitter.com/tink>
>
Re: how to reset kafka offset in zookeeper
Posted by Jens Rantil <je...@tink.se>.
Hi,
I noticed that a consumer in the new consumer API supports setting the
offset for a partition to beginning. I assume doing so also would update
the offset in Zookeeper eventually.
Cheers,
Jens
On Friday, December 18, 2015, Akhilesh Pathodia <pa...@gmail.com>
wrote:
> Hi,
>
> I want to reset the kafka offset in zookeeper so that the consumer will
> start reading messages from first offset. I am using flume as a consumer to
> kafka. I have set the kafka property kafka.auto.offset.reset to "smallest",
> but it does not reset the offset in zookeeper and that's why flume will not
> read messages from first offset.
>
> Is there any way to reset kafka offset in zookeeper?
>
> Thanks,
> Akhilesh
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>