any way to expire tuples after some time

[Mohsen Bande <mo...@gmail.com>]

Hi Everyone
i need tuples in streams to expire if there is no task to process them for
a while
is there any way to implement this in storm concepts?
i'm looking for a way to expire tuples when they are waiting for a bolt to
process them.
when incoming input tuples rate is higher than bolt processing rate, there
is some sort of queue that holds tuples until a bolt take them, right?
i want to expire waiting tuples after some time,
any help would be highly appreciated

Re: any way to expire tuples after some time

[Jacob Johansen <jo...@gmail.com>]

for 1.0.x you could reduce the size of
the TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE and change then change
the BACKPRESSURE_DISRUPTOR_HIGH_WATERMARK
and BACKPRESSURE_DISRUPTOR_LOW_WATERMARK.

or you could throttle it by rebalancing spouts and
using TOPOLOGY_MAX_SPOUT_PENDING this method will only work in reliable
mode I believe.

Jacob Johansen

On Tue, Nov 8, 2016 at 7:43 AM, Andrew Anderson <aa...@jsitelecom.com>
wrote:

> You could probably include a timestamp as part of your tuples, and simply
> reject any tuples at the bolt level if you find their timestamp is too old.
>
>
>
> *From:* Mohsen Bande [mailto:mohsenbande@gmail.com]
> *Sent:* Monday, November 07, 2016 1:51 AM
> *To:* user@storm.apache.org
> *Subject:* any way to expire tuples after some time
>
>
>
> Hi Everyone
>
> i need tuples in streams to expire if there is no task to process them for
> a while
> is there any way to implement this in storm concepts?
>
> i'm looking for a way to expire tuples when they are waiting for a bolt to
> process them.
> when incoming input tuples rate is higher than bolt processing rate, there
> is some sort of queue that holds tuples until a bolt take them, right?
> i want to expire waiting tuples after some time,
>
> any help would be highly appreciated
>

RE: any way to expire tuples after some time

[Andrew Anderson <aa...@jsitelecom.com>]

You could probably include a timestamp as part of your tuples, and simply reject any tuples at the bolt level if you find their timestamp is too old.

From: Mohsen Bande [mailto:mohsenbande@gmail.com]
Sent: Monday, November 07, 2016 1:51 AM
To: user@storm.apache.org
Subject: any way to expire tuples after some time

Hi Everyone
i need tuples in streams to expire if there is no task to process them for a while
is there any way to implement this in storm concepts?
i'm looking for a way to expire tuples when they are waiting for a bolt to process them.
when incoming input tuples rate is higher than bolt processing rate, there is some sort of queue that holds tuples until a bolt take them, right?
i want to expire waiting tuples after some time,
any help would be highly appreciated