You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Subrat Basnet <su...@myktm.com> on 2015/06/02 14:04:07 UTC
long running bolts
Hi there,
Is it normal to have long running bolts once in a while? When I say long running, I’m talking about a bolt that takes a few hours to process a tuple.
I need to export data, push notifications and upload files with this when I reach the LAST tuple of a sequence of tuples. This does not happen on every tuple.
I am worried, that this will make my whole topology hang up. My instinct is to give this particular bolt a higher parallelism, so that other threads are available to process when one bolt is hung up.
Please advise, what would be the best way to achieve this.
Thanks!
Subrat
--
Subrat Basnet
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: long running bolts
Posted by Nathan Leung <nc...@gmail.com>.
I agree with Mike, I would send the long running tuple out of your topology
to be processed. The timeout must be long enough to handle your expected
worst case, and if most messages are very fast then having a few long
running tuples will mask any potential problems you have with your other
data.
-Nathan
On Jun 2, 2015 9:00 AM, "Subrat Basnet" <su...@myktm.com> wrote:
> Thanks Mike.
>
> What if I increase the parallelism of the long running bolt, so that other
> threads would still go thru it when export is not required?
> Will ONE bolt thread taking a long time, still need me to increase the
> message timeout to a very high number?
>
> I’ve been contemplating sending the long running bolt’s task to
> a German job queue asynchronously and have it process there instead of
> Storm.
>
>
> --
> Subrat Basnet
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, June 2, 2015 at 6:29 PM, Mike Thomsen wrote:
>
> Not that I know of. I don't think Storm has an effective mechanism for
> allowing a particular tuple to take such an unusually long time while not
> affecting the amount of time allotted to the other bolts and tuples. You
> could jack those numbers up very, very high, but that could conceivably
> encourage other bolts to take longer than you want instead of
> failing/replaying a tuple.
>
> If you REALLY want to use storm for this long processing of a single piece
> of data, you could do something like this supposing it's just one bolt that
> does the long processing:
>
> 1. Write the data to disk, hdfs, hbase, RDBMS, etc.
> 2. Write a new topology based on a signal spout (zookeeper signals)
> 3. Give your new topology a ridiculously high amount of time for
> processing a single tuple
> 4. Have your current topology use SignalClient to post a zookeeper message
> for the the new one, when the last tuple is ready to be processed
>
> On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <su...@myktm.com> wrote:
>
> Hi there,
>
> Is it normal to have long running bolts once in a while? When I say long
> running, I’m talking about a bolt that takes a few hours to process a tuple.
>
> I need to export data, push notifications and upload files with this when
> I reach the LAST tuple of a sequence of tuples. This does not happen on
> every tuple.
>
> I am worried, that this will make my whole topology hang up. My instinct
> is to give this particular bolt a higher parallelism, so that other threads
> are available to process when one bolt is hung up.
>
> Please advise, what would be the best way to achieve this.
>
> Thanks!
> Subrat
>
> --
> Subrat Basnet
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
>
>
>
Re: long running bolts
Posted by Nathan Leung <nc...@gmail.com>.
You can emit the long running tuple unanchored but if you need to guarantee
its processing then it should be sent to another topology or processing
framework.
In fact even if it's in the same topology unanchored I would create
separate instances of the bolt for these long running tuples. If you do
not do this, the long running tuple can block other tuples and cause them
to timeout.
For example, if you have 2 bolt tasks with shuffle grouping, then tuples
are always evenly distributed. If one is processing a long tuple, any other
tuples that go to it will be blocked. Same holds for fields grouping, and
other groupings, regardless of number of bolt tasks. However if you create
tasks for this bolt using the same class but different name in the
topology, you can route long running bolts (without ack) to the separate
instances and they will not affect your normal processing.
On Jun 2, 2015 9:10 AM, "Mike Thomsen" <mi...@gmail.com> wrote:
> The way I understand the documentation, the tuple will not be fully purged
> from the system until all bolts it can pass through have reported success
> back to the spout. That may not hold true with unanchored tuples, but
> should definitely be true for ones you've anchored (ie... emit(tuple,
> someMessageId) )
>
> You probably can do this with Storm, I just wouldn't mess with the current
> topology.
>
> On Tue, Jun 2, 2015 at 8:58 AM, Subrat Basnet <su...@myktm.com> wrote:
>
>> Thanks Mike.
>>
>> What if I increase the parallelism of the long running bolt, so that
>> other threads would still go thru it when export is not required?
>> Will ONE bolt thread taking a long time, still need me to increase the
>> message timeout to a very high number?
>>
>> I’ve been contemplating sending the long running bolt’s task to
>> a German job queue asynchronously and have it process there instead of
>> Storm.
>>
>>
>> --
>> Subrat Basnet
>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>>
>> On Tuesday, June 2, 2015 at 6:29 PM, Mike Thomsen wrote:
>>
>> Not that I know of. I don't think Storm has an effective mechanism for
>> allowing a particular tuple to take such an unusually long time while not
>> affecting the amount of time allotted to the other bolts and tuples. You
>> could jack those numbers up very, very high, but that could conceivably
>> encourage other bolts to take longer than you want instead of
>> failing/replaying a tuple.
>>
>> If you REALLY want to use storm for this long processing of a single
>> piece of data, you could do something like this supposing it's just one
>> bolt that does the long processing:
>>
>> 1. Write the data to disk, hdfs, hbase, RDBMS, etc.
>> 2. Write a new topology based on a signal spout (zookeeper signals)
>> 3. Give your new topology a ridiculously high amount of time for
>> processing a single tuple
>> 4. Have your current topology use SignalClient to post a zookeeper
>> message for the the new one, when the last tuple is ready to be processed
>>
>> On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <su...@myktm.com> wrote:
>>
>> Hi there,
>>
>> Is it normal to have long running bolts once in a while? When I say long
>> running, I’m talking about a bolt that takes a few hours to process a tuple.
>>
>> I need to export data, push notifications and upload files with this when
>> I reach the LAST tuple of a sequence of tuples. This does not happen on
>> every tuple.
>>
>> I am worried, that this will make my whole topology hang up. My instinct
>> is to give this particular bolt a higher parallelism, so that other threads
>> are available to process when one bolt is hung up.
>>
>> Please advise, what would be the best way to achieve this.
>>
>> Thanks!
>> Subrat
>>
>> --
>> Subrat Basnet
>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>>
>>
>>
>>
>
Re: long running bolts
Posted by Mike Thomsen <mi...@gmail.com>.
The way I understand the documentation, the tuple will not be fully purged
from the system until all bolts it can pass through have reported success
back to the spout. That may not hold true with unanchored tuples, but
should definitely be true for ones you've anchored (ie... emit(tuple,
someMessageId) )
You probably can do this with Storm, I just wouldn't mess with the current
topology.
On Tue, Jun 2, 2015 at 8:58 AM, Subrat Basnet <su...@myktm.com> wrote:
> Thanks Mike.
>
> What if I increase the parallelism of the long running bolt, so that other
> threads would still go thru it when export is not required?
> Will ONE bolt thread taking a long time, still need me to increase the
> message timeout to a very high number?
>
> I’ve been contemplating sending the long running bolt’s task to
> a German job queue asynchronously and have it process there instead of
> Storm.
>
>
> --
> Subrat Basnet
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, June 2, 2015 at 6:29 PM, Mike Thomsen wrote:
>
> Not that I know of. I don't think Storm has an effective mechanism for
> allowing a particular tuple to take such an unusually long time while not
> affecting the amount of time allotted to the other bolts and tuples. You
> could jack those numbers up very, very high, but that could conceivably
> encourage other bolts to take longer than you want instead of
> failing/replaying a tuple.
>
> If you REALLY want to use storm for this long processing of a single piece
> of data, you could do something like this supposing it's just one bolt that
> does the long processing:
>
> 1. Write the data to disk, hdfs, hbase, RDBMS, etc.
> 2. Write a new topology based on a signal spout (zookeeper signals)
> 3. Give your new topology a ridiculously high amount of time for
> processing a single tuple
> 4. Have your current topology use SignalClient to post a zookeeper message
> for the the new one, when the last tuple is ready to be processed
>
> On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <su...@myktm.com> wrote:
>
> Hi there,
>
> Is it normal to have long running bolts once in a while? When I say long
> running, I’m talking about a bolt that takes a few hours to process a tuple.
>
> I need to export data, push notifications and upload files with this when
> I reach the LAST tuple of a sequence of tuples. This does not happen on
> every tuple.
>
> I am worried, that this will make my whole topology hang up. My instinct
> is to give this particular bolt a higher parallelism, so that other threads
> are available to process when one bolt is hung up.
>
> Please advise, what would be the best way to achieve this.
>
> Thanks!
> Subrat
>
> --
> Subrat Basnet
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
>
>
>
Re: long running bolts
Posted by Subrat Basnet <su...@myktm.com>.
Thanks Mike.
What if I increase the parallelism of the long running bolt, so that other threads would still go thru it when export is not required?
Will ONE bolt thread taking a long time, still need me to increase the message timeout to a very high number?
I’ve been contemplating sending the long running bolt’s task to a German job queue asynchronously and have it process there instead of Storm.
--
Subrat Basnet
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, June 2, 2015 at 6:29 PM, Mike Thomsen wrote:
> Not that I know of. I don't think Storm has an effective mechanism for allowing a particular tuple to take such an unusually long time while not affecting the amount of time allotted to the other bolts and tuples. You could jack those numbers up very, very high, but that could conceivably encourage other bolts to take longer than you want instead of failing/replaying a tuple.
>
> If you REALLY want to use storm for this long processing of a single piece of data, you could do something like this supposing it's just one bolt that does the long processing:
>
> 1. Write the data to disk, hdfs, hbase, RDBMS, etc.
> 2. Write a new topology based on a signal spout (zookeeper signals)
> 3. Give your new topology a ridiculously high amount of time for processing a single tuple
> 4. Have your current topology use SignalClient to post a zookeeper message for the the new one, when the last tuple is ready to be processed
>
> On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <subrat@myktm.com (mailto:subrat@myktm.com)> wrote:
> > Hi there,
> >
> > Is it normal to have long running bolts once in a while? When I say long running, I’m talking about a bolt that takes a few hours to process a tuple.
> >
> > I need to export data, push notifications and upload files with this when I reach the LAST tuple of a sequence of tuples. This does not happen on every tuple.
> >
> > I am worried, that this will make my whole topology hang up. My instinct is to give this particular bolt a higher parallelism, so that other threads are available to process when one bolt is hung up.
> >
> > Please advise, what would be the best way to achieve this.
> >
> > Thanks!
> > Subrat
> >
> > --
> > Subrat Basnet
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
Re: long running bolts
Posted by Mike Thomsen <mi...@gmail.com>.
Not that I know of. I don't think Storm has an effective mechanism for
allowing a particular tuple to take such an unusually long time while not
affecting the amount of time allotted to the other bolts and tuples. You
could jack those numbers up very, very high, but that could conceivably
encourage other bolts to take longer than you want instead of
failing/replaying a tuple.
If you REALLY want to use storm for this long processing of a single piece
of data, you could do something like this supposing it's just one bolt that
does the long processing:
1. Write the data to disk, hdfs, hbase, RDBMS, etc.
2. Write a new topology based on a signal spout (zookeeper signals)
3. Give your new topology a ridiculously high amount of time for processing
a single tuple
4. Have your current topology use SignalClient to post a zookeeper message
for the the new one, when the last tuple is ready to be processed
On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <su...@myktm.com> wrote:
> Hi there,
>
> Is it normal to have long running bolts once in a while? When I say long
> running, I’m talking about a bolt that takes a few hours to process a tuple.
>
> I need to export data, push notifications and upload files with this when
> I reach the LAST tuple of a sequence of tuples. This does not happen on
> every tuple.
>
> I am worried, that this will make my whole topology hang up. My instinct
> is to give this particular bolt a higher parallelism, so that other threads
> are available to process when one bolt is hung up.
>
> Please advise, what would be the best way to achieve this.
>
> Thanks!
> Subrat
>
> --
> Subrat Basnet
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
>