You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Pradheep s <pr...@gmail.com> on 2015/06/28 17:51:00 UTC
Block spout
Hi,
I have a spout which is emitting random numbers continuously to a bolt
which received them. Is it possible to block the spout for a finite time
from sending the tuples to the bolt? i have to send a blocking signal from
the bolt to spout?if possible how to do it?
Please let me know if someone can give some about this.
Thanks,
Pradheep
Re: Block spout
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
I don't know the "storm signals" project you mentioned. It might work
for you. It is not part of storm-core system.
However, I do not understand why you write to a file and read from it
again. Why not using one more bolt and just process the tuples there?
I would also not sleep the spout in your case, but only sync
writing/reading from file.
-Matthias
On 06/29/2015 04:00 PM, Pradheep s wrote:
> Hi,
> The problem i have is that, i write the tuples received form a spout to a
> file through a bolt. I also use another thread to read form it. Sometime
> the writer might overtake reader and override tuple which the reader has to
> read and lose tuples. So i have to avoid this..
> I heard about two things, i have to give back the control from bolt to
> spout in order for spout to sent the next tuple. is this right or the spot
> continuously sends the tuples whatever the bolt does with it?
> Next is about storm signals https://github.com/ptgoetz/storm-signals, is
> this feature available , so that i can try to send a sleep signal to spout
> whenever i need?
>
> Thanks,
> Pradheep
>
> On Mon, Jun 29, 2015 at 5:16 AM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
>> It depends what you want so accomplish... You can always "sleep" in
>> Spout.nextTuple() to block the spout for a finite time.
>>
>> As an alternative, you can limit the number of pending tuples be setting
>> parameter "topology.max.spout.pending" (be aware that tuples might time
>> out in this case).
>>
>> Sending a signal from bolt to spout is not supported by Storm. If you
>> want to do this, you need to code it by yourself. But it is tricky to
>> do, I would not recommend it.
>>
>>
>> -Matthias
>>
>>
>> On 06/28/2015 05:51 PM, Pradheep s wrote:
>>> Hi,
>>>
>>> I have a spout which is emitting random numbers continuously to a bolt
>>> which received them. Is it possible to block the spout for a finite time
>>> from sending the tuples to the bolt? i have to send a blocking signal
>> from
>>> the bolt to spout?if possible how to do it?
>>> Please let me know if someone can give some about this.
>>>
>>> Thanks,
>>> Pradheep
>>>
>>
>>
>
Re: Block spout
Posted by Ravi Sharma <pi...@gmail.com>.
Hi Pradeep,
It seems you want to send one tuple from spout to Bolt then bolt will write
it to a file and some other cosumer/bolt will read from file. And while
consumer reading, other tuple can be received by bolt and may write more
content on the same file.
is this right? if yes then continue reading.
this can be achived by ack/fail feature.
Once you send a tuple from spout, set a flag in spout(must be serializable)
as paused. So everytime Spout is called to get next tuple check this flag
if its paused return null
Now When bolt has processed the tuple it can send back ack/fail and in
spout when you receive this ack/fail, you can reset the flag to active.
Now Spout can send next message but you may still have problem of
overwriting as reader may be really slow. In that case you will have to
take care of it at Bolt level by checking somehow if consumer has consumed
the whole file.
*Little feedback :*
Not sure why you are writing to a file. Consider file as data stream from
one process/thread/bolt to other, Storm has already given really good
streaming system so why not use that.?
Ravi.
On Mon, Jun 29, 2015 at 3:00 PM, Pradheep s <pr...@gmail.com> wrote:
> Hi,
> The problem i have is that, i write the tuples received form a spout to a
> file through a bolt. I also use another thread to read form it. Sometime
> the writer might overtake reader and override tuple which the reader has to
> read and lose tuples. So i have to avoid this..
> I heard about two things, i have to give back the control from bolt to
> spout in order for spout to sent the next tuple. is this right or the spot
> continuously sends the tuples whatever the bolt does with it?
> Next is about storm signals https://github.com/ptgoetz/storm-signals, is
> this feature available , so that i can try to send a sleep signal to spout
> whenever i need?
>
> Thanks,
> Pradheep
>
> On Mon, Jun 29, 2015 at 5:16 AM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
> > It depends what you want so accomplish... You can always "sleep" in
> > Spout.nextTuple() to block the spout for a finite time.
> >
> > As an alternative, you can limit the number of pending tuples be setting
> > parameter "topology.max.spout.pending" (be aware that tuples might time
> > out in this case).
> >
> > Sending a signal from bolt to spout is not supported by Storm. If you
> > want to do this, you need to code it by yourself. But it is tricky to
> > do, I would not recommend it.
> >
> >
> > -Matthias
> >
> >
> > On 06/28/2015 05:51 PM, Pradheep s wrote:
> > > Hi,
> > >
> > > I have a spout which is emitting random numbers continuously to a bolt
> > > which received them. Is it possible to block the spout for a finite
> time
> > > from sending the tuples to the bolt? i have to send a blocking signal
> > from
> > > the bolt to spout?if possible how to do it?
> > > Please let me know if someone can give some about this.
> > >
> > > Thanks,
> > > Pradheep
> > >
> >
> >
>
Re: Block spout
Posted by Pradheep s <pr...@gmail.com>.
Hi,
The problem i have is that, i write the tuples received form a spout to a
file through a bolt. I also use another thread to read form it. Sometime
the writer might overtake reader and override tuple which the reader has to
read and lose tuples. So i have to avoid this..
I heard about two things, i have to give back the control from bolt to
spout in order for spout to sent the next tuple. is this right or the spot
continuously sends the tuples whatever the bolt does with it?
Next is about storm signals https://github.com/ptgoetz/storm-signals, is
this feature available , so that i can try to send a sleep signal to spout
whenever i need?
Thanks,
Pradheep
On Mon, Jun 29, 2015 at 5:16 AM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> It depends what you want so accomplish... You can always "sleep" in
> Spout.nextTuple() to block the spout for a finite time.
>
> As an alternative, you can limit the number of pending tuples be setting
> parameter "topology.max.spout.pending" (be aware that tuples might time
> out in this case).
>
> Sending a signal from bolt to spout is not supported by Storm. If you
> want to do this, you need to code it by yourself. But it is tricky to
> do, I would not recommend it.
>
>
> -Matthias
>
>
> On 06/28/2015 05:51 PM, Pradheep s wrote:
> > Hi,
> >
> > I have a spout which is emitting random numbers continuously to a bolt
> > which received them. Is it possible to block the spout for a finite time
> > from sending the tuples to the bolt? i have to send a blocking signal
> from
> > the bolt to spout?if possible how to do it?
> > Please let me know if someone can give some about this.
> >
> > Thanks,
> > Pradheep
> >
>
>
Re: Block spout
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
It depends what you want so accomplish... You can always "sleep" in
Spout.nextTuple() to block the spout for a finite time.
As an alternative, you can limit the number of pending tuples be setting
parameter "topology.max.spout.pending" (be aware that tuples might time
out in this case).
Sending a signal from bolt to spout is not supported by Storm. If you
want to do this, you need to code it by yourself. But it is tricky to
do, I would not recommend it.
-Matthias
On 06/28/2015 05:51 PM, Pradheep s wrote:
> Hi,
>
> I have a spout which is emitting random numbers continuously to a bolt
> which received them. Is it possible to block the spout for a finite time
> from sending the tuples to the bolt? i have to send a blocking signal from
> the bolt to spout?if possible how to do it?
> Please let me know if someone can give some about this.
>
> Thanks,
> Pradheep
>
Re: Block spout
Posted by 임정택 <ka...@gmail.com>.
No, currently there's no way to leave a message from downstream to
upstream.
Other tricky way is sharing external storage (like ZK) and setting a flag
value manually.
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2015년 6월 29일 월요일, Pradheep s<pr...@gmail.com>님이 작성한 메시지:
> Hi,
>
> I have a spout which is emitting random numbers continuously to a bolt
> which received them. Is it possible to block the spout for a finite time
> from sending the tuples to the bolt? i have to send a blocking signal from
> the bolt to spout?if possible how to do it?
> Please let me know if someone can give some about this.
>
> Thanks,
> Pradheep
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior