Backpressure limitations

[Owen Rees-Hayward <ow...@agilesoft.co.uk>]

Hi,

I noticed that the current back pressure implementation stops upstream
spouts even if the triggering bolt has 'free' instances (due to
parallelism) and is on a stream that has shuffle grouping (rather than by
field).

Conceptually, it seems that this is unnecessary and that, with shuffle
grouping, Storm could still distribute tuples over the bolts that were not
exhibiting back pressure.

I was wondering if anyone was aware of any work to improve this for either
1.x or the 2.0 work?

Thanks, Owen

-- 
*Owen Rees-Hayward*
*Data Engineer*
owenrh.me.uk
owen@agilesoft.co.uk
+44 (0) 7912 876046

*follow me on twitter?* <http://twitter.com/owen4d>

Re: Backpressure limitations

[Jacob Johansen <jo...@gmail.com>]

Theres a back pressure bug that is supposed to be fixed in 1.0.3 (not
released yet as far as I know)

currently we are using max pending from spout to throttle amount of
messages.


On January 19, 2017 at 5:12:00 AM, Owen Rees-Hayward (owen@agilesoft.co.uk)
wrote:

Hi,

I noticed that the current back pressure implementation stops upstream
spouts even if the triggering bolt has 'free' instances (due to
parallelism) and is on a stream that has shuffle grouping (rather than by
field).

Conceptually, it seems that this is unnecessary and that, with shuffle
grouping, Storm could still distribute tuples over the bolts that were not
exhibiting back pressure.

I was wondering if anyone was aware of any work to improve this for either
1.x or the 2.0 work?

Thanks, Owen

--
*Owen Rees-Hayward*
*Data Engineer*
owenrh.me.uk
owen@agilesoft.co.uk
+44 (0) 7912 876046

*follow me on twitter?* <http://twitter.com/owen4d>