You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marc Limotte <ml...@feeva.com> on 2009/04/30 02:25:37 UTC
Specifying System Properties in the had
I'm trying to set a System Property in the Hadoop config, so my jobs will know which cluster they are running on. I think I should be able to do this with -Dname=value in "mapred.child.java.opts" (example below), but the setting is ignored.
In hadoop-site.xml I have:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m -Dfoo=test</value>
</property>
But the job conf through the web server indicates:
mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC
I'm using Hadoop-0.17.2.1.
Any tips on why my setting is not picked up?
Marc
________________________________
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
Re: Specifying System Properties in the had
Posted by Tom White <to...@cloudera.com>.
Another way to do this would be to set a property in the Hadoop config itself.
In the job launcher you would have something like:
JobConf conf = ...
conf.setProperty("foo", "test");
Then you can read the property in your map or reduce task.
Tom
On Thu, Apr 30, 2009 at 3:25 PM, Aaron Kimball <aa...@cloudera.com> wrote:
> So you want a different "-Dfoo=test" on each node? It's probably grabbing
> the setting from the node where the job was submitted, and this overrides
> the settings on each task node.
>
> Try adding <final>true</final> to the <property> block on the tasktrackers,
> then restart Hadoop and try again. This will prevent the job from overriding
> the setting.
>
> - Aaron
>
> On Thu, Apr 30, 2009 at 9:25 AM, Marc Limotte <ml...@feeva.com> wrote:
>
>> I'm trying to set a System Property in the Hadoop config, so my jobs will
>> know which cluster they are running on. I think I should be able to do this
>> with -Dname=value in "mapred.child.java.opts" (example below), but the
>> setting is ignored.
>> In hadoop-site.xml I have:
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx200m -Dfoo=test</value>
>> </property>
>> But the job conf through the web server indicates:
>> mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC
>>
>> I'm using Hadoop-0.17.2.1.
>> Any tips on why my setting is not picked up?
>>
>> Marc
>>
>> ________________________________
>> PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
>> ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
>> PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
>> DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
>> PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
>> PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
>>
>
Re: Specifying System Properties in the had
Posted by Aaron Kimball <aa...@cloudera.com>.
So you want a different "-Dfoo=test" on each node? It's probably grabbing
the setting from the node where the job was submitted, and this overrides
the settings on each task node.
Try adding <final>true</final> to the <property> block on the tasktrackers,
then restart Hadoop and try again. This will prevent the job from overriding
the setting.
- Aaron
On Thu, Apr 30, 2009 at 9:25 AM, Marc Limotte <ml...@feeva.com> wrote:
> I'm trying to set a System Property in the Hadoop config, so my jobs will
> know which cluster they are running on. I think I should be able to do this
> with -Dname=value in "mapred.child.java.opts" (example below), but the
> setting is ignored.
> In hadoop-site.xml I have:
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx200m -Dfoo=test</value>
> </property>
> But the job conf through the web server indicates:
> mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC
>
> I'm using Hadoop-0.17.2.1.
> Any tips on why my setting is not picked up?
>
> Marc
>
> ________________________________
> PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
> ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
> PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
> DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
> PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
> PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
>