You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Shao, Saisai" <sa...@intel.com> on 2014/07/17 04:08:15 UTC
Question about ConsoleConsumer
Hi folks,
A newbie question when investigating ConsoleConsumer's code,
When we use ConsoleConsumer by setting "-from-beginning", we will set property as "auto.offset.reset"="smallest", which means moving the offset to the beginning of the data. But in the code, we still need to delete Zookeeper's group metadata as:
if(options.has(resetBeginningOpt))
ZkUtils.maybeDeletePath(options.valueOf(zkConnectOpt), "/consumers/" + options.valueOf(groupIdOpt))
>From my understanding, setting "auto.offset.rest"="smallest" is enough to seek to the beginning of the data, why we still need to delete Zookeeper metadata? Is there any specific reason or some under-the-cover concerns?
Sorry for the unthoughtful question and really appreciate your answer.
Thanks
Jerry
RE: Question about ConsoleConsumer
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Joe,
Thanks for your answer, seems I know what's its purpose.
Thanks a lot.
Jerry
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Thursday, July 17, 2014 10:15 AM
To: dev@kafka.apache.org
Cc: users@kafka.apache.org
Subject: Re: Question about ConsoleConsumer
auto.offset.reset is only when there is not an already established offset to consume from or an offset that no longer exists (and is invalid) in the case of truncated logs or compaction).
You can force this behavior by deleting the znode
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Wed, Jul 16, 2014 at 10:08 PM, Shao, Saisai <sa...@intel.com>
wrote:
> Hi folks,
>
> A newbie question when investigating ConsoleConsumer's code,
>
> When we use ConsoleConsumer by setting "-from-beginning", we will set
> property as "auto.offset.reset"="smallest", which means moving the
> offset to the beginning of the data. But in the code, we still need to
> delete Zookeeper's group metadata as:
>
> if(options.has(resetBeginningOpt))
> ZkUtils.maybeDeletePath(options.valueOf(zkConnectOpt), "/consumers/"
> + options.valueOf(groupIdOpt))
>
> From my understanding, setting "auto.offset.rest"="smallest" is enough
> to seek to the beginning of the data, why we still need to delete
> Zookeeper metadata? Is there any specific reason or some under-the-cover concerns?
>
> Sorry for the unthoughtful question and really appreciate your answer.
>
> Thanks
> Jerry
>
RE: Question about ConsoleConsumer
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Joe,
Thanks for your answer, seems I know what's its purpose.
Thanks a lot.
Jerry
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Thursday, July 17, 2014 10:15 AM
To: dev@kafka.apache.org
Cc: users@kafka.apache.org
Subject: Re: Question about ConsoleConsumer
auto.offset.reset is only when there is not an already established offset to consume from or an offset that no longer exists (and is invalid) in the case of truncated logs or compaction).
You can force this behavior by deleting the znode
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Wed, Jul 16, 2014 at 10:08 PM, Shao, Saisai <sa...@intel.com>
wrote:
> Hi folks,
>
> A newbie question when investigating ConsoleConsumer's code,
>
> When we use ConsoleConsumer by setting "-from-beginning", we will set
> property as "auto.offset.reset"="smallest", which means moving the
> offset to the beginning of the data. But in the code, we still need to
> delete Zookeeper's group metadata as:
>
> if(options.has(resetBeginningOpt))
> ZkUtils.maybeDeletePath(options.valueOf(zkConnectOpt), "/consumers/"
> + options.valueOf(groupIdOpt))
>
> From my understanding, setting "auto.offset.rest"="smallest" is enough
> to seek to the beginning of the data, why we still need to delete
> Zookeeper metadata? Is there any specific reason or some under-the-cover concerns?
>
> Sorry for the unthoughtful question and really appreciate your answer.
>
> Thanks
> Jerry
>
Re: Question about ConsoleConsumer
Posted by Joe Stein <jo...@stealth.ly>.
auto.offset.reset is only when there is not an already established offset
to consume from or an offset that no longer exists (and is invalid) in the
case of truncated logs or compaction).
You can force this behavior by deleting the znode
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Wed, Jul 16, 2014 at 10:08 PM, Shao, Saisai <sa...@intel.com>
wrote:
> Hi folks,
>
> A newbie question when investigating ConsoleConsumer's code,
>
> When we use ConsoleConsumer by setting "-from-beginning", we will set
> property as "auto.offset.reset"="smallest", which means moving the offset
> to the beginning of the data. But in the code, we still need to delete
> Zookeeper's group metadata as:
>
> if(options.has(resetBeginningOpt))
> ZkUtils.maybeDeletePath(options.valueOf(zkConnectOpt), "/consumers/"
> + options.valueOf(groupIdOpt))
>
> From my understanding, setting "auto.offset.rest"="smallest" is enough to
> seek to the beginning of the data, why we still need to delete Zookeeper
> metadata? Is there any specific reason or some under-the-cover concerns?
>
> Sorry for the unthoughtful question and really appreciate your answer.
>
> Thanks
> Jerry
>
Re: Question about ConsoleConsumer
Posted by Joe Stein <jo...@stealth.ly>.
auto.offset.reset is only when there is not an already established offset
to consume from or an offset that no longer exists (and is invalid) in the
case of truncated logs or compaction).
You can force this behavior by deleting the znode
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Wed, Jul 16, 2014 at 10:08 PM, Shao, Saisai <sa...@intel.com>
wrote:
> Hi folks,
>
> A newbie question when investigating ConsoleConsumer's code,
>
> When we use ConsoleConsumer by setting "-from-beginning", we will set
> property as "auto.offset.reset"="smallest", which means moving the offset
> to the beginning of the data. But in the code, we still need to delete
> Zookeeper's group metadata as:
>
> if(options.has(resetBeginningOpt))
> ZkUtils.maybeDeletePath(options.valueOf(zkConnectOpt), "/consumers/"
> + options.valueOf(groupIdOpt))
>
> From my understanding, setting "auto.offset.rest"="smallest" is enough to
> seek to the beginning of the data, why we still need to delete Zookeeper
> metadata? Is there any specific reason or some under-the-cover concerns?
>
> Sorry for the unthoughtful question and really appreciate your answer.
>
> Thanks
> Jerry
>