You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Jason Kania <ja...@ymail.com.INVALID> on 2016/06/17 01:14:58 UTC
Timeout increase on spout
For a media analysis application that I am working on, I need the ability to increase the tuple timeout dynamically since the analysis can take a short or long while and I don't want to have to set the timeout for the worst possible case as that will lead to long pauses in the event of failure. I am thinking to use a simple heartbeat mechanism that adds another time increment periodically such that in the event that a failure occurs, the timeout can still happen within a short time frame.
I would like to implement this in something common like the kafka spout, but was wondering if there are some existing communication channels back to the spout that could be enhanced for this purpose or if I would have to add a whole listening thread to do so.
Any pointers in this direction would be appreciated.
Thanks,
Jason
Re: Timeout increase on spout
Posted by Abhishek Agarwal <ab...@gmail.com>.
STORM-1549 - https://github.com/apache/storm/pull/1174
On Fri, Jun 17, 2016 at 9:10 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> I thought someone else was working on that already, but I cannot find the
> JIRA/Pull request for it. Does anyone else remember this?
>
> - Bobby
>
> On Thursday, June 16, 2016 8:18 PM, Jason Kania
> <ja...@ymail.com.INVALID> wrote:
>
>
> For a media analysis application that I am working on, I need the ability
> to increase the tuple timeout dynamically since the analysis can take a
> short or long while and I don't want to have to set the timeout for the
> worst possible case as that will lead to long pauses in the event of
> failure. I am thinking to use a simple heartbeat mechanism that adds
> another time increment periodically such that in the event that a failure
> occurs, the timeout can still happen within a short time frame.
> I would like to implement this in something common like the kafka spout,
> but was wondering if there are some existing communication channels back to
> the spout that could be enhanced for this purpose or if I would have to add
> a whole listening thread to do so.
> Any pointers in this direction would be appreciated.
> Thanks,
> Jason
>
>
>
>
--
Regards,
Abhishek Agarwal
Re: Timeout increase on spout
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
I thought someone else was working on that already, but I cannot find the JIRA/Pull request for it. Does anyone else remember this?
- Bobby
On Thursday, June 16, 2016 8:18 PM, Jason Kania <ja...@ymail.com.INVALID> wrote:
For a media analysis application that I am working on, I need the ability to increase the tuple timeout dynamically since the analysis can take a short or long while and I don't want to have to set the timeout for the worst possible case as that will lead to long pauses in the event of failure. I am thinking to use a simple heartbeat mechanism that adds another time increment periodically such that in the event that a failure occurs, the timeout can still happen within a short time frame.
I would like to implement this in something common like the kafka spout, but was wondering if there are some existing communication channels back to the spout that could be enhanced for this purpose or if I would have to add a whole listening thread to do so.
Any pointers in this direction would be appreciated.
Thanks,
Jason
Re: Timeout increase on spout
Posted by Abhishek Agarwal <ab...@gmail.com>.
Through your bolt, you can reset the timeout for an input tuple.
On Jun 17, 2016 6:47 AM, "Jason Kania" <ja...@ymail.com.invalid>
wrote:
For a media analysis application that I am working on, I need the ability
to increase the tuple timeout dynamically since the analysis can take a
short or long while and I don't want to have to set the timeout for the
worst possible case as that will lead to long pauses in the event of
failure. I am thinking to use a simple heartbeat mechanism that adds
another time increment periodically such that in the event that a failure
occurs, the timeout can still happen within a short time frame.
I would like to implement this in something common like the kafka spout,
but was wondering if there are some existing communication channels back to
the spout that could be enhanced for this purpose or if I would have to add
a whole listening thread to do so.
Any pointers in this direction would be appreciated.
Thanks,
Jason