You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com> on 2013/04/15 13:10:46 UTC
Adjusting tasktracker heap size?
Hi,
I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?
Thank you,
Marcos
Re: Adjusting tasktracker heap size?
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Thank you for the replies. Thankfully, this cluster works with a fairly regular load, so it shouldn't be too hard to fine-tune.
Regards,
Marcos
On 17-04-2013 09:23, Bejoy Ks wrote:
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com>> wrote:
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Hi,
I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?
Thank you,
Marcos
Re: Adjusting tasktracker heap size?
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Thank you for the replies. Thankfully, this cluster works with a fairly regular load, so it shouldn't be too hard to fine-tune.
Regards,
Marcos
On 17-04-2013 09:23, Bejoy Ks wrote:
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com>> wrote:
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Hi,
I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?
Thank you,
Marcos
Re: Adjusting tasktracker heap size?
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Thank you for the replies. Thankfully, this cluster works with a fairly regular load, so it shouldn't be too hard to fine-tune.
Regards,
Marcos
On 17-04-2013 09:23, Bejoy Ks wrote:
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com>> wrote:
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Hi,
I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?
Thank you,
Marcos
Re: Adjusting tasktracker heap size?
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Thank you for the replies. Thankfully, this cluster works with a fairly regular load, so it shouldn't be too hard to fine-tune.
Regards,
Marcos
On 17-04-2013 09:23, Bejoy Ks wrote:
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com>> wrote:
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Hi,
I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?
Thank you,
Marcos
Re: Adjusting tasktracker heap size?
Posted by Bejoy Ks <be...@gmail.com>.
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons
like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say
if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com> wrote:
> It depends on the type of job that is frequently submitting.
> RAM size of the machine.
> Heap size of tasktracker= (mapslots+reduceslots)*jvm size
> We can adjust this according to our requirement to fine tune our cluster.
> This is my thought.
>
>
> On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Hi,
>>
>> I am currently tuning a cluster, and I haven't found much information on
>> what factors to consider while adjusting the heap size of tasktrackers.
>> Is it a direct multiple of the number of map+reduce slots? Is there
>> anything else I should consider?
>>
>> Thank you,
>> Marcos
>
>
>
Re: Adjusting tasktracker heap size?
Posted by Bejoy Ks <be...@gmail.com>.
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons
like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say
if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com> wrote:
> It depends on the type of job that is frequently submitting.
> RAM size of the machine.
> Heap size of tasktracker= (mapslots+reduceslots)*jvm size
> We can adjust this according to our requirement to fine tune our cluster.
> This is my thought.
>
>
> On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Hi,
>>
>> I am currently tuning a cluster, and I haven't found much information on
>> what factors to consider while adjusting the heap size of tasktrackers.
>> Is it a direct multiple of the number of map+reduce slots? Is there
>> anything else I should consider?
>>
>> Thank you,
>> Marcos
>
>
>
Re: Adjusting tasktracker heap size?
Posted by Bejoy Ks <be...@gmail.com>.
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons
like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say
if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com> wrote:
> It depends on the type of job that is frequently submitting.
> RAM size of the machine.
> Heap size of tasktracker= (mapslots+reduceslots)*jvm size
> We can adjust this according to our requirement to fine tune our cluster.
> This is my thought.
>
>
> On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Hi,
>>
>> I am currently tuning a cluster, and I haven't found much information on
>> what factors to consider while adjusting the heap size of tasktrackers.
>> Is it a direct multiple of the number of map+reduce slots? Is there
>> anything else I should consider?
>>
>> Thank you,
>> Marcos
>
>
>
Re: Adjusting tasktracker heap size?
Posted by Bejoy Ks <be...@gmail.com>.
Hi Marcos,
You need to consider the slots based on the available memory
Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons
like DN,TT + Memory for other servicess if any running in that node)
Now you need to consider the generic MR jobs planned on your cluster. Say
if your tasks need 1G of JVM to run gracefully, then
Possible number of slots = Available Memory / JVM size of each task
Now divide the slots between mappers and reducers.
On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <am...@gmail.com> wrote:
> It depends on the type of job that is frequently submitting.
> RAM size of the machine.
> Heap size of tasktracker= (mapslots+reduceslots)*jvm size
> We can adjust this according to our requirement to fine tune our cluster.
> This is my thought.
>
>
> On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Hi,
>>
>> I am currently tuning a cluster, and I haven't found much information on
>> what factors to consider while adjusting the heap size of tasktrackers.
>> Is it a direct multiple of the number of map+reduce slots? Is there
>> anything else I should consider?
>>
>> Thank you,
>> Marcos
>
>
>
Re: Adjusting tasktracker heap size?
Posted by Amal G Jose <am...@gmail.com>.
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Hi,
>
> I am currently tuning a cluster, and I haven't found much information on
> what factors to consider while adjusting the heap size of tasktrackers.
> Is it a direct multiple of the number of map+reduce slots? Is there
> anything else I should consider?
>
> Thank you,
> Marcos
Re: Adjusting tasktracker heap size?
Posted by Amal G Jose <am...@gmail.com>.
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Hi,
>
> I am currently tuning a cluster, and I haven't found much information on
> what factors to consider while adjusting the heap size of tasktrackers.
> Is it a direct multiple of the number of map+reduce slots? Is there
> anything else I should consider?
>
> Thank you,
> Marcos
Re: Adjusting tasktracker heap size?
Posted by Amal G Jose <am...@gmail.com>.
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Hi,
>
> I am currently tuning a cluster, and I haven't found much information on
> what factors to consider while adjusting the heap size of tasktrackers.
> Is it a direct multiple of the number of map+reduce slots? Is there
> anything else I should consider?
>
> Thank you,
> Marcos
Re: Adjusting tasktracker heap size?
Posted by Amal G Jose <am...@gmail.com>.
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.
On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Hi,
>
> I am currently tuning a cluster, and I haven't found much information on
> what factors to consider while adjusting the heap size of tasktrackers.
> Is it a direct multiple of the number of map+reduce slots? Is there
> anything else I should consider?
>
> Thank you,
> Marcos