You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Eranga Heshan <er...@gmail.com> on 2016/12/08 02:07:24 UTC
Worker's Behavior With Heap Limit
Hi all,
In Storm, I configured the worker's maximum heap size to 1900MB. Then I
carefully monitored what happens when it reaches the limit.
The worker is killed and a new worker is spawned with another process id.
Is my observation correct? If so why is that happening?
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
Re: Worker's Behavior With Heap Limit
Posted by Ambud Sharma <as...@gmail.com>.
LRU caches are an effective memory management technique for Storm bolts if
lookup is what you are trying to do however if you are doing in memory
aggregations, I highly recommend sticking with standard Java maps and then
checkpoint state to an external data store (hbase, redis etc.)
Note:
* Storm aggregation MUST be time bound else you will be out of memory
* In Storm community we are trying to use Caffeine for Storm instead of
Guava.
On Thu, Dec 8, 2016 at 1:53 AM, Matt Lowe <ma...@gmail.com> wrote:
> Depending on what your memory needs to do you might find Guava useful:
> https://github.com/google/guava/wiki/CachesExplained
>
> Its a time/access based cache which will remove elements if they time out.
>
> Its all dependant on what your business logic is though.
>
>
> On 8 December 2016 at 10:48:17, Eranga Heshan (eranga.h.n@gmail.com)
> wrote:
>
> I think you are correct because I ran "free -m" command on the terminal
> and saw that free memory was rapidly decreasing. When it was near empty,
> the worker was killed. Then a new worker was started with a new PID.
>
> And yeah, there is a lot of data cached in memory waiting to be executed
> or waiting to be sent over the network. Probably that would be the issue.
>
> I have not heard of Guava caching technique. Instead of caching, I think
> of extending the memory of the server so I could extend the heap assignment
> to the worker.
>
> Thanks,
> Regards,
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Thu, Dec 8, 2016 at 2:42 PM, Matt Lowe <ma...@gmail.com>
> wrote:
>
>> My guess is the following:
>>
>> 1. Memory limit is reached within worker JVM
>> 2. Worker process starts to throw OutOfMemoryErrors (though this could be
>> silent and not in the logs)
>> 3. JVM will terminate due to lack of memory
>> 4. Supervisor will start a new JVM with new process Id and start a worker
>>
>> This will happen over and over. As storm is a Highly Available service, a
>> new worker will be added if one should die.
>>
>> I assume you are caching a lot of data in your bolt to reach this limit?
>> Have you tried using Guava cache?
>>
>> //Matt
>>
>>
>> On 8 December 2016 at 09:19:43, Matt Lowe (matthewneillowe@gmail.com)
>> wrote:
>>
>> Hello Eranga,
>>
>> Do you have the worker/supervisor logs when the worker dies?
>>
>> //Matt
>>
>>
>> On 8 December 2016 at 03:07:39, Eranga Heshan (eranga.h.n@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>> In Storm, I configured the worker's maximum heap size to 1900MB. Then I
>> carefully monitored what happens when it reaches the limit.
>>
>> The worker is killed and a new worker is spawned with another process id.
>> Is my observation correct? If so why is that happening?
>>
>> Thanks,
>> Regards,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>>
>
Re: Worker's Behavior With Heap Limit
Posted by Matt Lowe <ma...@gmail.com>.
Depending on what your memory needs to do you might find Guava useful:
https://github.com/google/guava/wiki/CachesExplained
Its a time/access based cache which will remove elements if they time out.
Its all dependant on what your business logic is though.
On 8 December 2016 at 10:48:17, Eranga Heshan (eranga.h.n@gmail.com) wrote:
I think you are correct because I ran "free -m" command on the terminal and
saw that free memory was rapidly decreasing. When it was near empty, the
worker was killed. Then a new worker was started with a new PID.
And yeah, there is a lot of data cached in memory waiting to be executed or
waiting to be sent over the network. Probably that would be the issue.
I have not heard of Guava caching technique. Instead of caching, I think of
extending the memory of the server so I could extend the heap assignment to
the worker.
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 8, 2016 at 2:42 PM, Matt Lowe <ma...@gmail.com> wrote:
> My guess is the following:
>
> 1. Memory limit is reached within worker JVM
> 2. Worker process starts to throw OutOfMemoryErrors (though this could be
> silent and not in the logs)
> 3. JVM will terminate due to lack of memory
> 4. Supervisor will start a new JVM with new process Id and start a worker
>
> This will happen over and over. As storm is a Highly Available service, a
> new worker will be added if one should die.
>
> I assume you are caching a lot of data in your bolt to reach this limit?
> Have you tried using Guava cache?
>
> //Matt
>
>
> On 8 December 2016 at 09:19:43, Matt Lowe (matthewneillowe@gmail.com)
> wrote:
>
> Hello Eranga,
>
> Do you have the worker/supervisor logs when the worker dies?
>
> //Matt
>
>
> On 8 December 2016 at 03:07:39, Eranga Heshan (eranga.h.n@gmail.com)
> wrote:
>
> Hi all,
>
> In Storm, I configured the worker's maximum heap size to 1900MB. Then I
> carefully monitored what happens when it reaches the limit.
>
> The worker is killed and a new worker is spawned with another process id.
> Is my observation correct? If so why is that happening?
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
>
Re: Worker's Behavior With Heap Limit
Posted by Eranga Heshan <er...@gmail.com>.
I think you are correct because I ran "free -m" command on the terminal and
saw that free memory was rapidly decreasing. When it was near empty, the
worker was killed. Then a new worker was started with a new PID.
And yeah, there is a lot of data cached in memory waiting to be executed or
waiting to be sent over the network. Probably that would be the issue.
I have not heard of Guava caching technique. Instead of caching, I think of
extending the memory of the server so I could extend the heap assignment to
the worker.
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 8, 2016 at 2:42 PM, Matt Lowe <ma...@gmail.com> wrote:
> My guess is the following:
>
> 1. Memory limit is reached within worker JVM
> 2. Worker process starts to throw OutOfMemoryErrors (though this could be
> silent and not in the logs)
> 3. JVM will terminate due to lack of memory
> 4. Supervisor will start a new JVM with new process Id and start a worker
>
> This will happen over and over. As storm is a Highly Available service, a
> new worker will be added if one should die.
>
> I assume you are caching a lot of data in your bolt to reach this limit?
> Have you tried using Guava cache?
>
> //Matt
>
>
> On 8 December 2016 at 09:19:43, Matt Lowe (matthewneillowe@gmail.com)
> wrote:
>
> Hello Eranga,
>
> Do you have the worker/supervisor logs when the worker dies?
>
> //Matt
>
>
> On 8 December 2016 at 03:07:39, Eranga Heshan (eranga.h.n@gmail.com)
> wrote:
>
> Hi all,
>
> In Storm, I configured the worker's maximum heap size to 1900MB. Then I
> carefully monitored what happens when it reaches the limit.
>
> The worker is killed and a new worker is spawned with another process id.
> Is my observation correct? If so why is that happening?
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
>
Re: Worker's Behavior With Heap Limit
Posted by Matt Lowe <ma...@gmail.com>.
My guess is the following:
1. Memory limit is reached within worker JVM
2. Worker process starts to throw OutOfMemoryErrors (though this could be
silent and not in the logs)
3. JVM will terminate due to lack of memory
4. Supervisor will start a new JVM with new process Id and start a worker
This will happen over and over. As storm is a Highly Available service, a
new worker will be added if one should die.
I assume you are caching a lot of data in your bolt to reach this limit?
Have you tried using Guava cache?
//Matt
On 8 December 2016 at 09:19:43, Matt Lowe (matthewneillowe@gmail.com) wrote:
Hello Eranga,
Do you have the worker/supervisor logs when the worker dies?
//Matt
On 8 December 2016 at 03:07:39, Eranga Heshan (eranga.h.n@gmail.com) wrote:
Hi all,
In Storm, I configured the worker's maximum heap size to 1900MB. Then I
carefully monitored what happens when it reaches the limit.
The worker is killed and a new worker is spawned with another process id.
Is my observation correct? If so why is that happening?
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
Re: Worker's Behavior With Heap Limit
Posted by Matt Lowe <ma...@gmail.com>.
Hello Eranga,
Do you have the worker/supervisor logs when the worker dies?
//Matt
On 8 December 2016 at 03:07:39, Eranga Heshan (eranga.h.n@gmail.com) wrote:
Hi all,
In Storm, I configured the worker's maximum heap size to 1900MB. Then I
carefully monitored what happens when it reaches the limit.
The worker is killed and a new worker is spawned with another process id.
Is my observation correct? If so why is that happening?
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>