You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Neha Goel <ne...@myntra.com> on 2017/10/04 18:05:01 UTC
Information on Back Pressure and Buffer Size
Hi,
I have a topology with almost 10 bolts. All the bolts calls different rest
services, and in the end the data is stored in redis and SOLR.
The problem happens with the back pressure, whenever one of the services
becomes slow, the whole topology takes a lot of time in coming back to its
speed. We read data from RabbitMQ. For 20Lakh messages, currently it is
taking around 4 hours.
Can you please suggest me on how to tune the back pressure.
Thanks,
Neha
Re: Information on Back Pressure and Buffer Size
Posted by "M. Aaron Bossert" <ma...@gmail.com>.
Just a thought, but you might also look at message timeout values as well as making your calls to the rest services asynchronous.
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Ali Nazemian <al...@gmail.com>
Sent: Thursday, October 5, 2017 7:50:39 AM
To: user@storm.apache.org
Subject: Re: Information on Back Pressure and Buffer Size
Since this version of Storm comes with automatic back pressure, you need to tune low water mark and high water mark parameters based on your application. Please have a look at the following guide it can help you to understand the way it works and help you to tune it based on your application.
http://jobs.one2team.com/apache-storms/
Cheers,
Ali
On Thu, Oct 5, 2017 at 10:31 PM, Neha Goel <ne...@myntra.com>> wrote:
I am using version 1.0.2.
On Thu, Oct 5, 2017 at 4:54 PM, Ali Nazemian <al...@gmail.com>> wrote:
Can I ask which version of Storm you are using? I wanted to understand whether it comes with automatic back pressure or not.
On 5 Oct. 2017 05:05, "Neha Goel" <ne...@myntra.com>> wrote:
Hi,
I have a topology with almost 10 bolts. All the bolts calls different rest services, and in the end the data is stored in redis and SOLR.
The problem happens with the back pressure, whenever one of the services becomes slow, the whole topology takes a lot of time in coming back to its speed. We read data from RabbitMQ. For 20Lakh messages, currently it is taking around 4 hours.
Can you please suggest me on how to tune the back pressure.
Thanks,
Neha
--
A.Nazemian
Re: Information on Back Pressure and Buffer Size
Posted by Ali Nazemian <al...@gmail.com>.
Since this version of Storm comes with automatic back pressure, you need to
tune low water mark and high water mark parameters based on your
application. Please have a look at the following guide it can help you to
understand the way it works and help you to tune it based on your
application.
http://jobs.one2team.com/apache-storms/
Cheers,
Ali
On Thu, Oct 5, 2017 at 10:31 PM, Neha Goel <ne...@myntra.com> wrote:
> I am using version 1.0.2.
>
> On Thu, Oct 5, 2017 at 4:54 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Can I ask which version of Storm you are using? I wanted to understand
>> whether it comes with automatic back pressure or not.
>>
>> On 5 Oct. 2017 05:05, "Neha Goel" <ne...@myntra.com> wrote:
>>
>>> Hi,
>>>
>>> I have a topology with almost 10 bolts. All the bolts calls different
>>> rest services, and in the end the data is stored in redis and SOLR.
>>> The problem happens with the back pressure, whenever one of the services
>>> becomes slow, the whole topology takes a lot of time in coming back to its
>>> speed. We read data from RabbitMQ. For 20Lakh messages, currently it is
>>> taking around 4 hours.
>>>
>>> Can you please suggest me on how to tune the back pressure.
>>>
>>> Thanks,
>>> Neha
>>>
>>
>
--
A.Nazemian
Re: Information on Back Pressure and Buffer Size
Posted by Neha Goel <ne...@myntra.com>.
I am using version 1.0.2.
On Thu, Oct 5, 2017 at 4:54 PM, Ali Nazemian <al...@gmail.com> wrote:
> Can I ask which version of Storm you are using? I wanted to understand
> whether it comes with automatic back pressure or not.
>
> On 5 Oct. 2017 05:05, "Neha Goel" <ne...@myntra.com> wrote:
>
>> Hi,
>>
>> I have a topology with almost 10 bolts. All the bolts calls different
>> rest services, and in the end the data is stored in redis and SOLR.
>> The problem happens with the back pressure, whenever one of the services
>> becomes slow, the whole topology takes a lot of time in coming back to its
>> speed. We read data from RabbitMQ. For 20Lakh messages, currently it is
>> taking around 4 hours.
>>
>> Can you please suggest me on how to tune the back pressure.
>>
>> Thanks,
>> Neha
>>
>
Re: Information on Back Pressure and Buffer Size
Posted by Ali Nazemian <al...@gmail.com>.
Can I ask which version of Storm you are using? I wanted to understand
whether it comes with automatic back pressure or not.
On 5 Oct. 2017 05:05, "Neha Goel" <ne...@myntra.com> wrote:
> Hi,
>
> I have a topology with almost 10 bolts. All the bolts calls different rest
> services, and in the end the data is stored in redis and SOLR.
> The problem happens with the back pressure, whenever one of the services
> becomes slow, the whole topology takes a lot of time in coming back to its
> speed. We read data from RabbitMQ. For 20Lakh messages, currently it is
> taking around 4 hours.
>
> Can you please suggest me on how to tune the back pressure.
>
> Thanks,
> Neha
>