How to Throttle in storm

[Gowtham S <go...@gmail.com>]

Hi,


We have certain bolts that invokes a few web services.  However, those
endpoints have limited throughput. So we wanted to find out if there any
recommendations on how to throttle the calls so that they don't overload
the downstream services.

Please let me know if there any hooks available in Storm, what are the
patterns that can be used and what are the best practices/pitfalls for
using them.

Thanks and regards,
Gowtham S, MCA
PH: 9597000673

Re: How to Throttle in storm

[Stephen Powis <sp...@salesforce.com>]

JVM based there are lots of examples online, but something like this:
https://www.baeldung.com/httpclient-connection-management
Where you set the maximum number of outbound connections to some limited
number.  If you have more bolts trying to execute requests against a pool
configured with fewer connections, your bolts would block until a
connection became available, essentially throttling?   Note: This breaks
down once you have bolts running on multiple JVMs, as each JVM would have
its own connection pool.

For external proxy service, which would work better when you have multiple
JVMs involved, you could use something like the apache proxy module
<https://stackoverflow.com/questions/14493574/configure-apache-web-server-to-use-a-proxy-server>.
I want to say it's possible to configure apache to restrict the number of
outbound connections to a specific set of hosts.  There may be better
software to solve this, it's just what jumps to mind for me.

On Mon, Nov 25, 2019 at 4:41 PM Gowtham S <go...@gmail.com> wrote:

> *Could you make use of an outbound HTTP connection pool? (FYI: Our bolts
> are span in multiple JVMs).*
>
> Is there any tool available for this? Can you please suggest if any
>
> With regards,
> Gowtham S
>
>
> On Mon, 25 Nov 2019 at 12:47, Stephen Powis <sp...@salesforce.com> wrote:
>
>> Could you make use of an outbound http connection pool?  Either via an
>> external proxy service your bolts talk through.... or some JVM based http
>> connection pool (which might get tricky if your bolts span multiple JVMs).
>>
>> On Mon, Nov 25, 2019 at 3:48 PM Gowtham S <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> We have certain bolts that invokes a few web services.  However, those
>>> endpoints have limited throughput. So we wanted to find out if there any
>>> recommendations on how to throttle the calls so that they don't overload
>>> the downstream services.
>>>
>>> Please let me know if there any hooks available in Storm, what are the
>>> patterns that can be used and what are the best practices/pitfalls for
>>> using them.
>>>
>>> Thanks and regards,
>>> Gowtham S, MCA
>>> PH: 9597000673
>>>
>>

Re: How to Throttle in storm

[Gowtham S <go...@gmail.com>]

*Could you make use of an outbound HTTP connection pool? (FYI: Our bolts
are span in multiple JVMs).*

Is there any tool available for this? Can you please suggest if any

With regards,
Gowtham S


On Mon, 25 Nov 2019 at 12:47, Stephen Powis <sp...@salesforce.com> wrote:

> Could you make use of an outbound http connection pool?  Either via an
> external proxy service your bolts talk through.... or some JVM based http
> connection pool (which might get tricky if your bolts span multiple JVMs).
>
> On Mon, Nov 25, 2019 at 3:48 PM Gowtham S <go...@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> We have certain bolts that invokes a few web services.  However, those
>> endpoints have limited throughput. So we wanted to find out if there any
>> recommendations on how to throttle the calls so that they don't overload
>> the downstream services.
>>
>> Please let me know if there any hooks available in Storm, what are the
>> patterns that can be used and what are the best practices/pitfalls for
>> using them.
>>
>> Thanks and regards,
>> Gowtham S, MCA
>> PH: 9597000673
>>
>

Re: How to Throttle in storm

[Stephen Powis <sp...@salesforce.com>]

Could you make use of an outbound http connection pool?  Either via an
external proxy service your bolts talk through.... or some JVM based http
connection pool (which might get tricky if your bolts span multiple JVMs).

On Mon, Nov 25, 2019 at 3:48 PM Gowtham S <go...@gmail.com> wrote:

> Hi,
>
>
> We have certain bolts that invokes a few web services.  However, those
> endpoints have limited throughput. So we wanted to find out if there any
> recommendations on how to throttle the calls so that they don't overload
> the downstream services.
>
> Please let me know if there any hooks available in Storm, what are the
> patterns that can be used and what are the best practices/pitfalls for
> using them.
>
> Thanks and regards,
> Gowtham S, MCA
> PH: 9597000673
>