You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Barry, Sean F" <se...@intel.com> on 2012/04/13 23:17:05 UTC
1gb allocated per thread for input read
*FYI this is a proof of concept cluster*
In my two node cluster that consists of
Master - Jobtracker, Datanode, Namenode, tasktracker, Secondarynamenode
And
Slave - Datenode , tasktraker
I have no more than 8g of ram on my slave and even less on the master and I am currently running 4 tasks on the slave and 2 on the master. My issue is that: is there a way where I can make sure that no more than 1g per thread is allocated to read a large input file for my job?
Thanks,
SB
Re: 1gb allocated per thread for input read
Posted by Serge Blazhievsky <Se...@nice.com>.
Yes.
Take a look at
http://hadoop.apache.org/common/docs/current/mapred-default.html
You can pass parameters via mapred.child.java.opts to task process and
each task will obey
Regards,
Serge Blazhievsky
On 4/13/12 2:41 PM, "Barry, Sean F" <se...@intel.com> wrote:
>Isnt essentially task = thread within my java app? Or maybe I am mistaken.
>
>My main concern is that I need an upper bound to limit the memory
>footprint for each java thread. Are there any configurations in Hadoop
>that can help me do so?
>
>-SB
>
>-----Original Message-----
>From: Serge Blazhievsky [mailto:Serge.Blazhiyevskyy@nice.com]
>Sent: Friday, April 13, 2012 2:20 PM
>To: common-user@hadoop.apache.org
>Subject: Re: 1gb allocated per thread for input read
>
>Per thread or per task?
>
>
>
>On 4/13/12 2:17 PM, "Barry, Sean F" <se...@intel.com> wrote:
>
>>*FYI this is a proof of concept cluster*
>>
>>In my two node cluster that consists of Master - Jobtracker, Datanode,
>>Namenode, tasktracker, Secondarynamenode And Slave - Datenode ,
>>tasktraker
>>
>>I have no more than 8g of ram on my slave and even less on the master
>>and I am currently running 4 tasks on the slave and 2 on the master. My
>>issue is that: is there a way where I can make sure that no more than
>>1g per thread is allocated to read a large input file for my job?
>>
>>Thanks,
>>SB
>
RE: 1gb allocated per thread for input read
Posted by "Barry, Sean F" <se...@intel.com>.
Isnt essentially task = thread within my java app? Or maybe I am mistaken.
My main concern is that I need an upper bound to limit the memory footprint for each java thread. Are there any configurations in Hadoop that can help me do so?
-SB
-----Original Message-----
From: Serge Blazhievsky [mailto:Serge.Blazhiyevskyy@nice.com]
Sent: Friday, April 13, 2012 2:20 PM
To: common-user@hadoop.apache.org
Subject: Re: 1gb allocated per thread for input read
Per thread or per task?
On 4/13/12 2:17 PM, "Barry, Sean F" <se...@intel.com> wrote:
>*FYI this is a proof of concept cluster*
>
>In my two node cluster that consists of Master - Jobtracker, Datanode,
>Namenode, tasktracker, Secondarynamenode And Slave - Datenode ,
>tasktraker
>
>I have no more than 8g of ram on my slave and even less on the master
>and I am currently running 4 tasks on the slave and 2 on the master. My
>issue is that: is there a way where I can make sure that no more than
>1g per thread is allocated to read a large input file for my job?
>
>Thanks,
>SB
Re: 1gb allocated per thread for input read
Posted by Serge Blazhievsky <Se...@nice.com>.
Per thread or per task?
On 4/13/12 2:17 PM, "Barry, Sean F" <se...@intel.com> wrote:
>*FYI this is a proof of concept cluster*
>
>In my two node cluster that consists of
>Master - Jobtracker, Datanode, Namenode, tasktracker, Secondarynamenode
>And
>Slave - Datenode , tasktraker
>
>I have no more than 8g of ram on my slave and even less on the master and
>I am currently running 4 tasks on the slave and 2 on the master. My issue
>is that: is there a way where I can make sure that no more than 1g per
>thread is allocated to read a large input file for my job?
>
>Thanks,
>SB