You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by "Rick.Magnuson" <Ri...@target.com> on 2017/07/03 18:31:38 UTC

Bolt Exponential Backoff?

I realize the Exponential Backoff Retry Manager exists for the KafkaSpout, but I’m wondering if anyone has implemented similar functionality in a Bolt?

I need to do this to ensure an ElasticSearch cluster isn’t overwhelmed with requests when reading from Kafka.

Rick Magnuson | Lead Data Engineer | •Target | 1000 Nicollet Mall | Minneapolis, MN 55403

Re: Bolt Exponential Backoff?

Posted by Srinath C <sr...@gmail.com>.

Hi Rick,

Not sure if you've already tried it, but you could use a ES bulk processor
[1] in the bolt to have a consistent indexing traffic into elasticsearch.
It's not based on Backoff but still prevents from overwhelming ES.

[1]
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html

Regards,
Srinath

On Jul 4, 2017 12:01 AM, "Rick.Magnuson" <Ri...@target.com> wrote:

> I realize the Exponential Backoff Retry Manager exists for the KafkaSpout,
> but I’m wondering if anyone has implemented similar functionality in a Bolt?
>
>
>
> I need to do this to ensure an ElasticSearch cluster isn’t overwhelmed
> with requests when reading from Kafka.
>
>
>
> Rick Magnuson | Lead Data Engineer | ¤Target | 1000 Nicollet Mall |
> Minneapolis, MN 55403
>
>
>