You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Fernando Padilla <fe...@alum.mit.edu> on 2009/07/22 05:41:23 UTC
best way to set memory
So.. I want to have different memory profiles for
NameNode/DataNode/JobTracker/TaskTracker.
But it looks like I only have one environment variable to modify,
HADOOP_HEAPSIZE, but I might be running more than one on a single
box/deployment/conf directory.
Is there a proper way to set the memory for each kind of server? Or has
an issue been created to document this bug/deficiency??
Re: best way to set memory
Posted by Fernando Padilla <fe...@alum.mit.edu>.
thank you :)
I already created an issue as well:
https://issues.apache.org/jira/browse/HADOOP-6167
and have a patch against it, maybe you can mark 6168 as a duplicate of 6167.
what do you think of the patch on 6167?
Allen Wittenauer wrote:
> FWIW, we actually push a completely separate config to the name node, jt,
> etc, because of some of the other settings (like saves and
> dfs.[in|ex]cludes). But if you wanted to do an all-in-one, well...
>
> Hmm. Looking at the code, this worked differently than I always thought it
> did (at least in 0.18). Like Amogh, I thought that HADOOP_NAMENODE_OPTS (or
> at least HADOOP_NAMENODE_HEAPSIZE) would override, but that clearly isn't
> the case.
>
> I've filed HADOOP-6168 and appropriately bonked some of my local Hadoop
> committers on the head. :)
>
> On 7/22/09 7:50 AM, "Fernando Padilla" <fe...@alum.mit.edu> wrote:
>
>> But right now the script forcefully adds and extra -Xmx1000m even if you
>> don't want it..
>>
>> I guess I'll be submitting a patch for hadoop-daemon.sh later. :) :)
>>
>> thank you all
>>
>>
>> On 7/22/09 2:25 AM, Amogh Vasekar wrote:
>>> I haven't played a lot with it, but you may want to check if setting
>>> HADOOP_NAMENODE_OPTS, HADOOP_TASKTRACKER_OPTS help. Let me know if you find a
>>> way to do this :)
>>>
>>> Cheers!
>>> Amogh
>>>
>>> -----Original Message-----
>>> From: Fernando Padilla [mailto:fern@alum.mit.edu]
>>> Sent: Wednesday, July 22, 2009 9:47 AM
>>> To: common-user@hadoop.apache.org
>>> Subject: Re: best way to set memory
>>>
>>> I was thinking not for M/R, but for the actual daemons:
>>>
>>> When I go and start up a daemon (like below). They all use the same
>>> hadoop-env.sh. Which allows you to only set the HADOOP_HEAPSIZE once..
>>> not differently for each daemon-type..
>>>
>>> bin/hadoop-daemon.sh start namenode
>>> bin/hadoop-daemon.sh start datanode
>>> bin/hadoop-daemon.sh start secondarynamenode
>>> bin/hadoop-daemon.sh start jobtracker
>>> bin/hadoop-daemon.sh start tasktracker
>>>
>>>
>>>
>>> Amogh Vasekar wrote:
>>>> If you need to set the java_options for mem., you can do this via configure
>>>> in your MR job.
>>>>
>>>> -----Original Message-----
>>>> From: Fernando Padilla [mailto:fern@alum.mit.edu]
>>>> Sent: Wednesday, July 22, 2009 9:11 AM
>>>> To: common-user@hadoop.apache.org
>>>> Subject: best way to set memory
>>>>
>>>> So.. I want to have different memory profiles for
>>>> NameNode/DataNode/JobTracker/TaskTracker.
>>>>
>>>> But it looks like I only have one environment variable to modify,
>>>> HADOOP_HEAPSIZE, but I might be running more than one on a single
>>>> box/deployment/conf directory.
>>>>
>>>> Is there a proper way to set the memory for each kind of server? Or has
>>>> an issue been created to document this bug/deficiency??
>
Re: best way to set memory
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
FWIW, we actually push a completely separate config to the name node, jt,
etc, because of some of the other settings (like saves and
dfs.[in|ex]cludes). But if you wanted to do an all-in-one, well...
Hmm. Looking at the code, this worked differently than I always thought it
did (at least in 0.18). Like Amogh, I thought that HADOOP_NAMENODE_OPTS (or
at least HADOOP_NAMENODE_HEAPSIZE) would override, but that clearly isn't
the case.
I've filed HADOOP-6168 and appropriately bonked some of my local Hadoop
committers on the head. :)
On 7/22/09 7:50 AM, "Fernando Padilla" <fe...@alum.mit.edu> wrote:
> But right now the script forcefully adds and extra -Xmx1000m even if you
> don't want it..
>
> I guess I'll be submitting a patch for hadoop-daemon.sh later. :) :)
>
> thank you all
>
>
> On 7/22/09 2:25 AM, Amogh Vasekar wrote:
>> I haven't played a lot with it, but you may want to check if setting
>> HADOOP_NAMENODE_OPTS, HADOOP_TASKTRACKER_OPTS help. Let me know if you find a
>> way to do this :)
>>
>> Cheers!
>> Amogh
>>
>> -----Original Message-----
>> From: Fernando Padilla [mailto:fern@alum.mit.edu]
>> Sent: Wednesday, July 22, 2009 9:47 AM
>> To: common-user@hadoop.apache.org
>> Subject: Re: best way to set memory
>>
>> I was thinking not for M/R, but for the actual daemons:
>>
>> When I go and start up a daemon (like below). They all use the same
>> hadoop-env.sh. Which allows you to only set the HADOOP_HEAPSIZE once..
>> not differently for each daemon-type..
>>
>> bin/hadoop-daemon.sh start namenode
>> bin/hadoop-daemon.sh start datanode
>> bin/hadoop-daemon.sh start secondarynamenode
>> bin/hadoop-daemon.sh start jobtracker
>> bin/hadoop-daemon.sh start tasktracker
>>
>>
>>
>> Amogh Vasekar wrote:
>>> If you need to set the java_options for mem., you can do this via configure
>>> in your MR job.
>>>
>>> -----Original Message-----
>>> From: Fernando Padilla [mailto:fern@alum.mit.edu]
>>> Sent: Wednesday, July 22, 2009 9:11 AM
>>> To: common-user@hadoop.apache.org
>>> Subject: best way to set memory
>>>
>>> So.. I want to have different memory profiles for
>>> NameNode/DataNode/JobTracker/TaskTracker.
>>>
>>> But it looks like I only have one environment variable to modify,
>>> HADOOP_HEAPSIZE, but I might be running more than one on a single
>>> box/deployment/conf directory.
>>>
>>> Is there a proper way to set the memory for each kind of server? Or has
>>> an issue been created to document this bug/deficiency??
Re: best way to set memory
Posted by Fernando Padilla <fe...@alum.mit.edu>.
But right now the script forcefully adds and extra -Xmx1000m even if you
don't want it..
I guess I'll be submitting a patch for hadoop-daemon.sh later. :) :)
thank you all
On 7/22/09 2:25 AM, Amogh Vasekar wrote:
> I haven't played a lot with it, but you may want to check if setting HADOOP_NAMENODE_OPTS, HADOOP_TASKTRACKER_OPTS help. Let me know if you find a way to do this :)
>
> Cheers!
> Amogh
>
> -----Original Message-----
> From: Fernando Padilla [mailto:fern@alum.mit.edu]
> Sent: Wednesday, July 22, 2009 9:47 AM
> To: common-user@hadoop.apache.org
> Subject: Re: best way to set memory
>
> I was thinking not for M/R, but for the actual daemons:
>
> When I go and start up a daemon (like below). They all use the same
> hadoop-env.sh. Which allows you to only set the HADOOP_HEAPSIZE once..
> not differently for each daemon-type..
>
> bin/hadoop-daemon.sh start namenode
> bin/hadoop-daemon.sh start datanode
> bin/hadoop-daemon.sh start secondarynamenode
> bin/hadoop-daemon.sh start jobtracker
> bin/hadoop-daemon.sh start tasktracker
>
>
>
> Amogh Vasekar wrote:
>> If you need to set the java_options for mem., you can do this via configure in your MR job.
>>
>> -----Original Message-----
>> From: Fernando Padilla [mailto:fern@alum.mit.edu]
>> Sent: Wednesday, July 22, 2009 9:11 AM
>> To: common-user@hadoop.apache.org
>> Subject: best way to set memory
>>
>> So.. I want to have different memory profiles for
>> NameNode/DataNode/JobTracker/TaskTracker.
>>
>> But it looks like I only have one environment variable to modify,
>> HADOOP_HEAPSIZE, but I might be running more than one on a single
>> box/deployment/conf directory.
>>
>> Is there a proper way to set the memory for each kind of server? Or has
>> an issue been created to document this bug/deficiency??
RE: best way to set memory
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
I haven't played a lot with it, but you may want to check if setting HADOOP_NAMENODE_OPTS, HADOOP_TASKTRACKER_OPTS help. Let me know if you find a way to do this :)
Cheers!
Amogh
-----Original Message-----
From: Fernando Padilla [mailto:fern@alum.mit.edu]
Sent: Wednesday, July 22, 2009 9:47 AM
To: common-user@hadoop.apache.org
Subject: Re: best way to set memory
I was thinking not for M/R, but for the actual daemons:
When I go and start up a daemon (like below). They all use the same
hadoop-env.sh. Which allows you to only set the HADOOP_HEAPSIZE once..
not differently for each daemon-type..
bin/hadoop-daemon.sh start namenode
bin/hadoop-daemon.sh start datanode
bin/hadoop-daemon.sh start secondarynamenode
bin/hadoop-daemon.sh start jobtracker
bin/hadoop-daemon.sh start tasktracker
Amogh Vasekar wrote:
> If you need to set the java_options for mem., you can do this via configure in your MR job.
>
> -----Original Message-----
> From: Fernando Padilla [mailto:fern@alum.mit.edu]
> Sent: Wednesday, July 22, 2009 9:11 AM
> To: common-user@hadoop.apache.org
> Subject: best way to set memory
>
> So.. I want to have different memory profiles for
> NameNode/DataNode/JobTracker/TaskTracker.
>
> But it looks like I only have one environment variable to modify,
> HADOOP_HEAPSIZE, but I might be running more than one on a single
> box/deployment/conf directory.
>
> Is there a proper way to set the memory for each kind of server? Or has
> an issue been created to document this bug/deficiency??
Re: best way to set memory
Posted by Fernando Padilla <fe...@alum.mit.edu>.
I was thinking not for M/R, but for the actual daemons:
When I go and start up a daemon (like below). They all use the same
hadoop-env.sh. Which allows you to only set the HADOOP_HEAPSIZE once..
not differently for each daemon-type..
bin/hadoop-daemon.sh start namenode
bin/hadoop-daemon.sh start datanode
bin/hadoop-daemon.sh start secondarynamenode
bin/hadoop-daemon.sh start jobtracker
bin/hadoop-daemon.sh start tasktracker
Amogh Vasekar wrote:
> If you need to set the java_options for mem., you can do this via configure in your MR job.
>
> -----Original Message-----
> From: Fernando Padilla [mailto:fern@alum.mit.edu]
> Sent: Wednesday, July 22, 2009 9:11 AM
> To: common-user@hadoop.apache.org
> Subject: best way to set memory
>
> So.. I want to have different memory profiles for
> NameNode/DataNode/JobTracker/TaskTracker.
>
> But it looks like I only have one environment variable to modify,
> HADOOP_HEAPSIZE, but I might be running more than one on a single
> box/deployment/conf directory.
>
> Is there a proper way to set the memory for each kind of server? Or has
> an issue been created to document this bug/deficiency??
RE: best way to set memory
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
If you need to set the java_options for mem., you can do this via configure in your MR job.
-----Original Message-----
From: Fernando Padilla [mailto:fern@alum.mit.edu]
Sent: Wednesday, July 22, 2009 9:11 AM
To: common-user@hadoop.apache.org
Subject: best way to set memory
So.. I want to have different memory profiles for
NameNode/DataNode/JobTracker/TaskTracker.
But it looks like I only have one environment variable to modify,
HADOOP_HEAPSIZE, but I might be running more than one on a single
box/deployment/conf directory.
Is there a proper way to set the memory for each kind of server? Or has
an issue been created to document this bug/deficiency??