You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by jinhong lu <ji...@gmail.com> on 2015/07/03 12:44:57 UTC
how to ensure storm not write message twice to local file
Hi,
I build a topo to get messages from by kafka, and then grep some keyword, if fit, write to local file.
I use OpaqueTridentKafkaSpout of storm-kafka to ensure the tuple will not miss or repeate, but consider one situation: when writing message to local file, and some error occurse(for example, not enough space). At this moment, some messages have writen to local file, and others not, if the spout resend the message, the message will be write twice.
how to do that?thanks.
jinhong_lu
Re: how to ensure storm not write message twice to local file
Posted by jinhong lu <ji...@gmail.com>.
Thanks Javier,
I want to built the topo like this:
read message from kafka—> grep —>persist the grep result to hdfs/base through Opaque state.
And then another stream(by drpc) to read message from hdfs/hbase, since the Opaque transaction keeps the pre state and current state, I can read message from the pre state.
Is it work? thanks.
> 在 2015年7月4日,02:21,Javier Gonzalez <ja...@gmail.com> 写道:
>
> Javier
Re: how to ensure storm not write message twice to local file
Posted by Javier Gonzalez <ja...@gmail.com>.
And of course, there's then the matter of the BOLT crashing and losing the
state, unless you keep it in a separate store, and... you get the idea.
On Fri, Jul 3, 2015 at 2:20 PM, Javier Gonzalez <ja...@gmail.com> wrote:
> You need to keep track of state in your bolt before writing. Yes, it is
> indeed quite a chore. For "exactly once", particularly when coordinating
> with external systems such as databases or output queues, Storm is, ah, not
> exactly the best fit. Unless keeping track of processed events to avoid
> duplication is feasible in your use case.
>
> Regards,
> Javier
>
> On Fri, Jul 3, 2015 at 6:44 AM, jinhong lu <ji...@gmail.com> wrote:
>
>> Hi,
>>
>> I build a topo to get messages from by kafka, and then grep some
>> keyword, if fit, write to local file.
>>
>> I use OpaqueTridentKafkaSpout of storm-kafka to ensure the tuple will not
>> miss or repeate, but consider one situation: when writing message to local
>> file, and some error occurse(for example, not enough space). At this
>> moment, some messages have writen to local file, and others not, if the
>> spout resend the message, the message will be write twice.
>>
>> how to do that?thanks.
>>
>> jinhong_lu
>>
>
>
>
> --
> Javier González Nicolini
>
--
Javier González Nicolini
Re: how to ensure storm not write message twice to local file
Posted by Javier Gonzalez <ja...@gmail.com>.
You need to keep track of state in your bolt before writing. Yes, it is
indeed quite a chore. For "exactly once", particularly when coordinating
with external systems such as databases or output queues, Storm is, ah, not
exactly the best fit. Unless keeping track of processed events to avoid
duplication is feasible in your use case.
Regards,
Javier
On Fri, Jul 3, 2015 at 6:44 AM, jinhong lu <ji...@gmail.com> wrote:
> Hi,
>
> I build a topo to get messages from by kafka, and then grep some
> keyword, if fit, write to local file.
>
> I use OpaqueTridentKafkaSpout of storm-kafka to ensure the tuple will not
> miss or repeate, but consider one situation: when writing message to local
> file, and some error occurse(for example, not enough space). At this
> moment, some messages have writen to local file, and others not, if the
> spout resend the message, the message will be write twice.
>
> how to do that?thanks.
>
> jinhong_lu
>
--
Javier González Nicolini