You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by 이승진 <sw...@navercorp.com> on 2014/12/01 10:48:27 UTC
is there a way for kafka spout to read from written zookeeper offset
Hi all,
Kafkaspout periodically write each partition offset to zookeeper.
and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from the latest offset.
Is there a option to read from last committed consumer offset?
When I restart topology due to bugfix or whatsoever, some of the messages are not processed by storm because I set it to -1.
Sincerely
Re: is there a way for kafka spout to read from written zookeeper offset
Posted by 이승진 <sw...@navercorp.com>.
thanks for clear answer, it really helped
-----Original Message-----
From: "Harsha"<storm@harsha.io>
To: <user@storm.apache.org>;
Cc:
Sent: 2014-12-02 (화) 22:51:29
Subject: Re: is there a way for kafka spout to read from written zookeeper offset
if you set spoutConfig.forceFromStart=true it will cause KafkaSpout to ignore the existing offsets stored in the zookeeper and starts reading from the beginning of the queue. It makes sense to have it in for the first time you deploy the topology and for the next runs ( kill and redeploy topology) make sure you turn off the forceFromStart.
On Mon, Dec 1, 2014, at 09:08 PM, 이승진 wrote:
thanks for comment.
even though I restart topology and obviously there exists offset information in zookeeper, "No partition information found, using configuration to determine offset" happens.
I'm using spoutConfig.forceFromStart = true; is this option has something to do with that? what exactly does that option mean?
-----Original Message-----
From: "Harsha"<storm@harsha.io>
To: <user@storm.apache.org>;
Cc:
Sent: 2014-12-02 (화) 02:27:58
Subject: Re: is there a way for kafka spout to read from written zookeeper offset
KafkaSpout will read last committed offset from zookeeper. If there is no offset in the zookeeper than only it will use configured startOffsetTime. Can you check in your worker logs after restarting your topology for this message "No partition information found, using configuration to determine offset"
-Harsha
On Mon, Dec 1, 2014, at 01:48 AM, 이승진 wrote:
Hi all,
Kafkaspout periodically write each partition offset to zookeeper.
and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from the latest offset.
Is there a option to read from last committed consumer offset?
When I restart topology due to bugfix or whatsoever, some of the messages are not processed by storm because I set it to -1.
Sincerely
Re: is there a way for kafka spout to read from written zookeeper offset
Posted by Harsha <st...@harsha.io>.
if you set spoutConfig.forceFromStart=true it will cause KafkaSpout to
ignore the existing offsets stored in the zookeeper and starts reading
from the beginning of the queue. It makes sense to have it in for the
first time you deploy the topology and for the next runs ( kill and
redeploy topology) make sure you turn off the forceFromStart.
On Mon, Dec 1, 2014, at 09:08 PM, 이승진 wrote:
> thanks for comment.
>
> even though I restart topology and obviously there exists offset
> information in zookeeper, "No partition information found, using
> configuration to determine offset" happens.
>
> I'm using spoutConfig.forceFromStart = true; is this option has
> something to do with that? what exactly does that option mean?
>
>
>
> -----Original Message----- *From:* "Harsha"<st...@harsha.io> *To:*
> <us...@storm.apache.org>; *Cc:* *Sent:* 2014-12-02 (화) 02:27:58
> *Subject:* Re: is there a way for kafka spout to read from written
> zookeeper offset
> KafkaSpout will read last committed offset from zookeeper. If there is
> no offset in the zookeeper than only it will use configured
> startOffsetTime. Can you check in your worker logs after restarting
> your topology for this message "No partition information found, using
> configuration to determine offset" -Harsha
>
>
> On Mon, Dec 1, 2014, at 01:48 AM, 이승진 wrote:
>> Hi all,
>>
>> Kafkaspout periodically write each partition offset to zookeeper.
>>
>> and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from
>> the latest offset.
>>
>> Is there a option to read from last committed consumer offset?
>>
>> When I restart topology due to bugfix or whatsoever, some of the
>> messages are not processed by storm because I set it to -1.
>>
>> Sincerely
>>
>
>
Re: is there a way for kafka spout to read from written zookeeper offset
Posted by 이승진 <sw...@navercorp.com>.
thanks for comment.
even though I restart topology and obviously there exists offset information in zookeeper, "No partition information found, using configuration to determine offset" happens.
I'm using spoutConfig.forceFromStart = true; is this option has something to do with that? what exactly does that option mean?
-----Original Message-----
From: "Harsha"<storm@harsha.io>
To: <user@storm.apache.org>;
Cc:
Sent: 2014-12-02 (화) 02:27:58
Subject: Re: is there a way for kafka spout to read from written zookeeper offset
KafkaSpout will read last committed offset from zookeeper. If there is no offset in the zookeeper than only it will use configured startOffsetTime. Can you check in your worker logs after restarting your topology for this message "No partition information found, using configuration to determine offset"
-Harsha
On Mon, Dec 1, 2014, at 01:48 AM, 이승진 wrote:
Hi all,
Kafkaspout periodically write each partition offset to zookeeper.
and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from the latest offset.
Is there a option to read from last committed consumer offset?
When I restart topology due to bugfix or whatsoever, some of the messages are not processed by storm because I set it to -1.
Sincerely
Re: is there a way for kafka spout to read from written zookeeper offset
Posted by Harsha <st...@harsha.io>.
KafkaSpout will read last committed offset from zookeeper. If there is
no offset in the zookeeper than only it will use configured
startOffsetTime. Can you check in your worker logs after restarting your
topology for this message "No partition information found, using
configuration to determine offset" -Harsha
On Mon, Dec 1, 2014, at 01:48 AM, 이승진 wrote:
> Hi all,
>
> Kafkaspout periodically write each partition offset to zookeeper.
>
> and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from
> the latest offset.
>
> Is there a option to read from last committed consumer offset?
>
> When I restart topology due to bugfix or whatsoever, some of the
> messages are not processed by storm because I set it to -1.
>
> Sincerely
>