You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Srinath C <sr...@gmail.com> on 2014/04/07 08:11:35 UTC
JVM heap options for Supervisor
Hi,
I'm trying to figure out a reasonable amount of heap to grant to the
supervisor process on the storm worker machines. What are the factors that
must be considered? I'm thinking - size of the topology jar and number of
slots should be considered. But would like to hear if anyone was able to
figure out the least amount of heap that can be safely granted.
Also, does the supervisor fully load the topology jar into memory or
are there any optimizations done?
Thanks,
Srinath.
Re: JVM heap options for Supervisor
Posted by Srinath C <sr...@gmail.com>.
Thanks Jason!
On Wed, Apr 9, 2014 at 11:10 AM, Jason Jackson <ja...@gmail.com> wrote:
> The supervisor calls downloadChunk on nimbus API
> https://github.com/apache/incubator-storm/blob/master/storm-core/src/storm.thrifteach chunk is persisted to local disk on the supervisor before the next
> chunk is grabbed. So the whole jar does need to be stored in RAM.
>
>
> On Tue, Apr 8, 2014 at 6:14 PM, Srinath C <sr...@gmail.com> wrote:
>
>> Thanks for the reply Jason.
>> Supervisor doesn't need it in the classpath. But to provide the classpath
>> to the worker processes it must be transferring the jar from nimbus. Hence
>> the question.
>>
>>
>> On Wed, Apr 9, 2014 at 6:32 AM, Jason Jackson <ja...@gmail.com>wrote:
>>
>>> supervisor JVM does not load topology jar in it's classpath. We use
>>> 256mb heapsize, that ought to be fine.
>>>
>>>
>>> On Sun, Apr 6, 2014 at 11:11 PM, Srinath C <sr...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I'm trying to figure out a reasonable amount of heap to grant to
>>>> the supervisor process on the storm worker machines. What are the factors
>>>> that must be considered? I'm thinking - size of the topology jar and number
>>>> of slots should be considered. But would like to hear if anyone was able to
>>>> figure out the least amount of heap that can be safely granted.
>>>>
>>>> Also, does the supervisor fully load the topology jar into memory
>>>> or are there any optimizations done?
>>>>
>>>> Thanks,
>>>> Srinath.
>>>>
>>>>
>>>
>>
>
Re: JVM heap options for Supervisor
Posted by Jason Jackson <ja...@gmail.com>.
The supervisor calls downloadChunk on nimbus API
https://github.com/apache/incubator-storm/blob/master/storm-core/src/storm.thrifteach
chunk is persisted to local disk on the supervisor before the next
chunk is grabbed. So the whole jar does need to be stored in RAM.
On Tue, Apr 8, 2014 at 6:14 PM, Srinath C <sr...@gmail.com> wrote:
> Thanks for the reply Jason.
> Supervisor doesn't need it in the classpath. But to provide the classpath
> to the worker processes it must be transferring the jar from nimbus. Hence
> the question.
>
>
> On Wed, Apr 9, 2014 at 6:32 AM, Jason Jackson <ja...@gmail.com> wrote:
>
>> supervisor JVM does not load topology jar in it's classpath. We use 256mb
>> heapsize, that ought to be fine.
>>
>>
>> On Sun, Apr 6, 2014 at 11:11 PM, Srinath C <sr...@gmail.com> wrote:
>>
>>> Hi,
>>> I'm trying to figure out a reasonable amount of heap to grant to the
>>> supervisor process on the storm worker machines. What are the factors that
>>> must be considered? I'm thinking - size of the topology jar and number of
>>> slots should be considered. But would like to hear if anyone was able to
>>> figure out the least amount of heap that can be safely granted.
>>>
>>> Also, does the supervisor fully load the topology jar into memory or
>>> are there any optimizations done?
>>>
>>> Thanks,
>>> Srinath.
>>>
>>>
>>
>
Re: JVM heap options for Supervisor
Posted by Srinath C <sr...@gmail.com>.
Thanks for the reply Jason.
Supervisor doesn't need it in the classpath. But to provide the classpath
to the worker processes it must be transferring the jar from nimbus. Hence
the question.
On Wed, Apr 9, 2014 at 6:32 AM, Jason Jackson <ja...@gmail.com> wrote:
> supervisor JVM does not load topology jar in it's classpath. We use 256mb
> heapsize, that ought to be fine.
>
>
> On Sun, Apr 6, 2014 at 11:11 PM, Srinath C <sr...@gmail.com> wrote:
>
>> Hi,
>> I'm trying to figure out a reasonable amount of heap to grant to the
>> supervisor process on the storm worker machines. What are the factors that
>> must be considered? I'm thinking - size of the topology jar and number of
>> slots should be considered. But would like to hear if anyone was able to
>> figure out the least amount of heap that can be safely granted.
>>
>> Also, does the supervisor fully load the topology jar into memory or
>> are there any optimizations done?
>>
>> Thanks,
>> Srinath.
>>
>>
>
Re: JVM heap options for Supervisor
Posted by Jason Jackson <ja...@gmail.com>.
supervisor JVM does not load topology jar in it's classpath. We use 256mb
heapsize, that ought to be fine.
On Sun, Apr 6, 2014 at 11:11 PM, Srinath C <sr...@gmail.com> wrote:
> Hi,
> I'm trying to figure out a reasonable amount of heap to grant to the
> supervisor process on the storm worker machines. What are the factors that
> must be considered? I'm thinking - size of the topology jar and number of
> slots should be considered. But would like to hear if anyone was able to
> figure out the least amount of heap that can be safely granted.
>
> Also, does the supervisor fully load the topology jar into memory or
> are there any optimizations done?
>
> Thanks,
> Srinath.
>
>