You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Vincent Y. Shen" <vi...@gmail.com> on 2013/11/14 15:53:43 UTC
mapred.tasktacker.reduce.tasks.maximum issue
Hi Guys,
I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
cluster. But regardless whatever I tried, hadoop refuses to use my value
but the default (2)
What I did do
===========
stop mapred
stop dfs
update the value of mapred.tasktacker.reduce.tasks.maximum
start dfs
start mapred
But the system still uses default values
Environment
=========
Hadoop version: 1.1.2
Java version: 7
Mapred-site.xml
============
<property>
<name>mapred.system.dir</name>
<value>/hadoop/mapred/system</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>50</value>
</property>
<property>
<name>mapred.tasktacker.reduce.tasks.maximum</name>
<value>15</value>
</property>
<property>
<name>mapred.tasktacker.map.tasks.maximum</name>
<value>15</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx500m</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>machine1.mydomain.com:54311</value>
</property>
Thanks guys for any pointers!!!!
Best regards,
Vincent.
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Adam Kawa <ka...@gmail.com>.
It looks that you have a typo in the names of configuration properties, so
Hadoop ignores them and uses the default vaules (2 map and 2 reduce tasks
per node).
it should mapred.*tasktracker*.reduce.tasks.maximum not
mapred.*tasktacker*.reduce.tasks.maximum
(tasktRacker, not tasktacker) - the same for map tasks.
2013/11/15 Zizon Qiu <zz...@gmail.com>
> may be it is not reading the right configuration file.
>
> if possible,make a malformed file,then restart tasktracker where it must
> fail to do so.
>
>
> On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen <
> vincent.y.shen@gmail.com> wrote:
>
>> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
>> are final
>>
>>
>> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> you could make the values final: add <final> true </final> to make sure
>>> they cannot be overridden..
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>>> Hi Dieter,
>>>>
>>>> Thanks a lot for prompt reply! very much appreciated.
>>>>
>>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>>> restarted the cluster 4 times but no luck...
>>>>
>>>>
>>>> Any other hint?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>>
>>>>> Dear Vincent,
>>>>>
>>>>> You have to make sure that mapred-site is modified on every
>>>>> tasktracker node! Not just the master! Mapred-site can be customized if you
>>>>> have a heterogeneous cluster. I usually start the dfs, then copy all the
>>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>>> job.
>>>>>
>>>>> Regards, Dieter
>>>>>
>>>>>
>>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>>
>>>>> Hi Guys,
>>>>>>
>>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>>> but the default (2)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What I did do
>>>>>> ===========
>>>>>>
>>>>>> stop mapred
>>>>>> stop dfs
>>>>>>
>>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>>
>>>>>>
>>>>>> start dfs
>>>>>> start mapred
>>>>>>
>>>>>>
>>>>>> But the system still uses default values
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Environment
>>>>>> =========
>>>>>>
>>>>>> Hadoop version: 1.1.2
>>>>>> Java version: 7
>>>>>>
>>>>>>
>>>>>> Mapred-site.xml
>>>>>> ============
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.system.dir</name>
>>>>>> <value>/hadoop/mapred/system</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.reduce.tasks</name>
>>>>>> <value>50</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.child.java.opts</name>
>>>>>> <value>-Xmx500m</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.job.tracker</name>
>>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks guys for any pointers!!!!
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>>
>>>>>> Vincent.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Adam Kawa <ka...@gmail.com>.
It looks that you have a typo in the names of configuration properties, so
Hadoop ignores them and uses the default vaules (2 map and 2 reduce tasks
per node).
it should mapred.*tasktracker*.reduce.tasks.maximum not
mapred.*tasktacker*.reduce.tasks.maximum
(tasktRacker, not tasktacker) - the same for map tasks.
2013/11/15 Zizon Qiu <zz...@gmail.com>
> may be it is not reading the right configuration file.
>
> if possible,make a malformed file,then restart tasktracker where it must
> fail to do so.
>
>
> On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen <
> vincent.y.shen@gmail.com> wrote:
>
>> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
>> are final
>>
>>
>> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> you could make the values final: add <final> true </final> to make sure
>>> they cannot be overridden..
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>>> Hi Dieter,
>>>>
>>>> Thanks a lot for prompt reply! very much appreciated.
>>>>
>>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>>> restarted the cluster 4 times but no luck...
>>>>
>>>>
>>>> Any other hint?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>>
>>>>> Dear Vincent,
>>>>>
>>>>> You have to make sure that mapred-site is modified on every
>>>>> tasktracker node! Not just the master! Mapred-site can be customized if you
>>>>> have a heterogeneous cluster. I usually start the dfs, then copy all the
>>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>>> job.
>>>>>
>>>>> Regards, Dieter
>>>>>
>>>>>
>>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>>
>>>>> Hi Guys,
>>>>>>
>>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>>> but the default (2)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What I did do
>>>>>> ===========
>>>>>>
>>>>>> stop mapred
>>>>>> stop dfs
>>>>>>
>>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>>
>>>>>>
>>>>>> start dfs
>>>>>> start mapred
>>>>>>
>>>>>>
>>>>>> But the system still uses default values
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Environment
>>>>>> =========
>>>>>>
>>>>>> Hadoop version: 1.1.2
>>>>>> Java version: 7
>>>>>>
>>>>>>
>>>>>> Mapred-site.xml
>>>>>> ============
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.system.dir</name>
>>>>>> <value>/hadoop/mapred/system</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.reduce.tasks</name>
>>>>>> <value>50</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.child.java.opts</name>
>>>>>> <value>-Xmx500m</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.job.tracker</name>
>>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks guys for any pointers!!!!
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>>
>>>>>> Vincent.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Adam Kawa <ka...@gmail.com>.
It looks that you have a typo in the names of configuration properties, so
Hadoop ignores them and uses the default vaules (2 map and 2 reduce tasks
per node).
it should mapred.*tasktracker*.reduce.tasks.maximum not
mapred.*tasktacker*.reduce.tasks.maximum
(tasktRacker, not tasktacker) - the same for map tasks.
2013/11/15 Zizon Qiu <zz...@gmail.com>
> may be it is not reading the right configuration file.
>
> if possible,make a malformed file,then restart tasktracker where it must
> fail to do so.
>
>
> On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen <
> vincent.y.shen@gmail.com> wrote:
>
>> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
>> are final
>>
>>
>> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> you could make the values final: add <final> true </final> to make sure
>>> they cannot be overridden..
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>>> Hi Dieter,
>>>>
>>>> Thanks a lot for prompt reply! very much appreciated.
>>>>
>>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>>> restarted the cluster 4 times but no luck...
>>>>
>>>>
>>>> Any other hint?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>>
>>>>> Dear Vincent,
>>>>>
>>>>> You have to make sure that mapred-site is modified on every
>>>>> tasktracker node! Not just the master! Mapred-site can be customized if you
>>>>> have a heterogeneous cluster. I usually start the dfs, then copy all the
>>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>>> job.
>>>>>
>>>>> Regards, Dieter
>>>>>
>>>>>
>>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>>
>>>>> Hi Guys,
>>>>>>
>>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>>> but the default (2)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What I did do
>>>>>> ===========
>>>>>>
>>>>>> stop mapred
>>>>>> stop dfs
>>>>>>
>>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>>
>>>>>>
>>>>>> start dfs
>>>>>> start mapred
>>>>>>
>>>>>>
>>>>>> But the system still uses default values
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Environment
>>>>>> =========
>>>>>>
>>>>>> Hadoop version: 1.1.2
>>>>>> Java version: 7
>>>>>>
>>>>>>
>>>>>> Mapred-site.xml
>>>>>> ============
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.system.dir</name>
>>>>>> <value>/hadoop/mapred/system</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.reduce.tasks</name>
>>>>>> <value>50</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.child.java.opts</name>
>>>>>> <value>-Xmx500m</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.job.tracker</name>
>>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks guys for any pointers!!!!
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>>
>>>>>> Vincent.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Adam Kawa <ka...@gmail.com>.
It looks that you have a typo in the names of configuration properties, so
Hadoop ignores them and uses the default vaules (2 map and 2 reduce tasks
per node).
it should mapred.*tasktracker*.reduce.tasks.maximum not
mapred.*tasktacker*.reduce.tasks.maximum
(tasktRacker, not tasktacker) - the same for map tasks.
2013/11/15 Zizon Qiu <zz...@gmail.com>
> may be it is not reading the right configuration file.
>
> if possible,make a malformed file,then restart tasktracker where it must
> fail to do so.
>
>
> On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen <
> vincent.y.shen@gmail.com> wrote:
>
>> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
>> are final
>>
>>
>> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> you could make the values final: add <final> true </final> to make sure
>>> they cannot be overridden..
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>>> Hi Dieter,
>>>>
>>>> Thanks a lot for prompt reply! very much appreciated.
>>>>
>>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>>> restarted the cluster 4 times but no luck...
>>>>
>>>>
>>>> Any other hint?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>>
>>>>> Dear Vincent,
>>>>>
>>>>> You have to make sure that mapred-site is modified on every
>>>>> tasktracker node! Not just the master! Mapred-site can be customized if you
>>>>> have a heterogeneous cluster. I usually start the dfs, then copy all the
>>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>>> job.
>>>>>
>>>>> Regards, Dieter
>>>>>
>>>>>
>>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>>
>>>>> Hi Guys,
>>>>>>
>>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>>> but the default (2)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What I did do
>>>>>> ===========
>>>>>>
>>>>>> stop mapred
>>>>>> stop dfs
>>>>>>
>>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>>
>>>>>>
>>>>>> start dfs
>>>>>> start mapred
>>>>>>
>>>>>>
>>>>>> But the system still uses default values
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Environment
>>>>>> =========
>>>>>>
>>>>>> Hadoop version: 1.1.2
>>>>>> Java version: 7
>>>>>>
>>>>>>
>>>>>> Mapred-site.xml
>>>>>> ============
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.system.dir</name>
>>>>>> <value>/hadoop/mapred/system</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.reduce.tasks</name>
>>>>>> <value>50</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>>> <value>15</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.child.java.opts</name>
>>>>>> <value>-Xmx500m</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>mapred.job.tracker</name>
>>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks guys for any pointers!!!!
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>>
>>>>>> Vincent.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Zizon Qiu <zz...@gmail.com>.
may be it is not reading the right configuration file.
if possible,make a malformed file,then restart tasktracker where it must
fail to do so.
On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen
<vi...@gmail.com>wrote:
> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
> are final
>
>
> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> you could make the values final: add <final> true </final> to make sure
>> they cannot be overridden..
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>>> Hi Dieter,
>>>
>>> Thanks a lot for prompt reply! very much appreciated.
>>>
>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>> restarted the cluster 4 times but no luck...
>>>
>>>
>>> Any other hint?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>
>>>> Dear Vincent,
>>>>
>>>> You have to make sure that mapred-site is modified on every tasktracker
>>>> node! Not just the master! Mapred-site can be customized if you have a
>>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>> job.
>>>>
>>>> Regards, Dieter
>>>>
>>>>
>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>
>>>> Hi Guys,
>>>>>
>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>> but the default (2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What I did do
>>>>> ===========
>>>>>
>>>>> stop mapred
>>>>> stop dfs
>>>>>
>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>
>>>>>
>>>>> start dfs
>>>>> start mapred
>>>>>
>>>>>
>>>>> But the system still uses default values
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Environment
>>>>> =========
>>>>>
>>>>> Hadoop version: 1.1.2
>>>>> Java version: 7
>>>>>
>>>>>
>>>>> Mapred-site.xml
>>>>> ============
>>>>>
>>>>> <property>
>>>>> <name>mapred.system.dir</name>
>>>>> <value>/hadoop/mapred/system</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.reduce.tasks</name>
>>>>> <value>50</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.child.java.opts</name>
>>>>> <value>-Xmx500m</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.job.tracker</name>
>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> Thanks guys for any pointers!!!!
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Vincent.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Zizon Qiu <zz...@gmail.com>.
may be it is not reading the right configuration file.
if possible,make a malformed file,then restart tasktracker where it must
fail to do so.
On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen
<vi...@gmail.com>wrote:
> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
> are final
>
>
> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> you could make the values final: add <final> true </final> to make sure
>> they cannot be overridden..
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>>> Hi Dieter,
>>>
>>> Thanks a lot for prompt reply! very much appreciated.
>>>
>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>> restarted the cluster 4 times but no luck...
>>>
>>>
>>> Any other hint?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>
>>>> Dear Vincent,
>>>>
>>>> You have to make sure that mapred-site is modified on every tasktracker
>>>> node! Not just the master! Mapred-site can be customized if you have a
>>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>> job.
>>>>
>>>> Regards, Dieter
>>>>
>>>>
>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>
>>>> Hi Guys,
>>>>>
>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>> but the default (2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What I did do
>>>>> ===========
>>>>>
>>>>> stop mapred
>>>>> stop dfs
>>>>>
>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>
>>>>>
>>>>> start dfs
>>>>> start mapred
>>>>>
>>>>>
>>>>> But the system still uses default values
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Environment
>>>>> =========
>>>>>
>>>>> Hadoop version: 1.1.2
>>>>> Java version: 7
>>>>>
>>>>>
>>>>> Mapred-site.xml
>>>>> ============
>>>>>
>>>>> <property>
>>>>> <name>mapred.system.dir</name>
>>>>> <value>/hadoop/mapred/system</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.reduce.tasks</name>
>>>>> <value>50</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.child.java.opts</name>
>>>>> <value>-Xmx500m</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.job.tracker</name>
>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> Thanks guys for any pointers!!!!
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Vincent.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Zizon Qiu <zz...@gmail.com>.
may be it is not reading the right configuration file.
if possible,make a malformed file,then restart tasktracker where it must
fail to do so.
On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen
<vi...@gmail.com>wrote:
> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
> are final
>
>
> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> you could make the values final: add <final> true </final> to make sure
>> they cannot be overridden..
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>>> Hi Dieter,
>>>
>>> Thanks a lot for prompt reply! very much appreciated.
>>>
>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>> restarted the cluster 4 times but no luck...
>>>
>>>
>>> Any other hint?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>
>>>> Dear Vincent,
>>>>
>>>> You have to make sure that mapred-site is modified on every tasktracker
>>>> node! Not just the master! Mapred-site can be customized if you have a
>>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>> job.
>>>>
>>>> Regards, Dieter
>>>>
>>>>
>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>
>>>> Hi Guys,
>>>>>
>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>> but the default (2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What I did do
>>>>> ===========
>>>>>
>>>>> stop mapred
>>>>> stop dfs
>>>>>
>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>
>>>>>
>>>>> start dfs
>>>>> start mapred
>>>>>
>>>>>
>>>>> But the system still uses default values
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Environment
>>>>> =========
>>>>>
>>>>> Hadoop version: 1.1.2
>>>>> Java version: 7
>>>>>
>>>>>
>>>>> Mapred-site.xml
>>>>> ============
>>>>>
>>>>> <property>
>>>>> <name>mapred.system.dir</name>
>>>>> <value>/hadoop/mapred/system</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.reduce.tasks</name>
>>>>> <value>50</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.child.java.opts</name>
>>>>> <value>-Xmx500m</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.job.tracker</name>
>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> Thanks guys for any pointers!!!!
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Vincent.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Zizon Qiu <zz...@gmail.com>.
may be it is not reading the right configuration file.
if possible,make a malformed file,then restart tasktracker where it must
fail to do so.
On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen
<vi...@gmail.com>wrote:
> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
> are final
>
>
> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> you could make the values final: add <final> true </final> to make sure
>> they cannot be overridden..
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>>> Hi Dieter,
>>>
>>> Thanks a lot for prompt reply! very much appreciated.
>>>
>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>> restarted the cluster 4 times but no luck...
>>>
>>>
>>> Any other hint?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>>
>>>> Dear Vincent,
>>>>
>>>> You have to make sure that mapred-site is modified on every tasktracker
>>>> node! Not just the master! Mapred-site can be customized if you have a
>>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>> job.
>>>>
>>>> Regards, Dieter
>>>>
>>>>
>>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>>
>>>> Hi Guys,
>>>>>
>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>> but the default (2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What I did do
>>>>> ===========
>>>>>
>>>>> stop mapred
>>>>> stop dfs
>>>>>
>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>
>>>>>
>>>>> start dfs
>>>>> start mapred
>>>>>
>>>>>
>>>>> But the system still uses default values
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Environment
>>>>> =========
>>>>>
>>>>> Hadoop version: 1.1.2
>>>>> Java version: 7
>>>>>
>>>>>
>>>>> Mapred-site.xml
>>>>> ============
>>>>>
>>>>> <property>
>>>>> <name>mapred.system.dir</name>
>>>>> <value>/hadoop/mapred/system</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.reduce.tasks</name>
>>>>> <value>50</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>> <value>15</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.child.java.opts</name>
>>>>> <value>-Xmx500m</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>> <name>mapred.job.tracker</name>
>>>>> <value>machine1.mydomain.com:54311</value>
>>>>>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> Thanks guys for any pointers!!!!
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Vincent.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
are final
On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com> wrote:
> you could make the values final: add <final> true </final> to make sure
> they cannot be overridden..
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
>> Hi Dieter,
>>
>> Thanks a lot for prompt reply! very much appreciated.
>>
>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>> restarted the cluster 4 times but no luck...
>>
>>
>> Any other hint?
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> Dear Vincent,
>>>
>>> You have to make sure that mapred-site is modified on every tasktracker
>>> node! Not just the master! Mapred-site can be customized if you have a
>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>> job.
>>>
>>> Regards, Dieter
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>> Hi Guys,
>>>>
>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>> but the default (2)
>>>>
>>>>
>>>>
>>>>
>>>> What I did do
>>>> ===========
>>>>
>>>> stop mapred
>>>> stop dfs
>>>>
>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>
>>>>
>>>> start dfs
>>>> start mapred
>>>>
>>>>
>>>> But the system still uses default values
>>>>
>>>>
>>>>
>>>>
>>>> Environment
>>>> =========
>>>>
>>>> Hadoop version: 1.1.2
>>>> Java version: 7
>>>>
>>>>
>>>> Mapred-site.xml
>>>> ============
>>>>
>>>> <property>
>>>> <name>mapred.system.dir</name>
>>>> <value>/hadoop/mapred/system</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.reduce.tasks</name>
>>>> <value>50</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.child.java.opts</name>
>>>> <value>-Xmx500m</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>machine1.mydomain.com:54311</value>
>>>>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Thanks guys for any pointers!!!!
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Vincent.
>>>>
>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
are final
On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com> wrote:
> you could make the values final: add <final> true </final> to make sure
> they cannot be overridden..
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
>> Hi Dieter,
>>
>> Thanks a lot for prompt reply! very much appreciated.
>>
>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>> restarted the cluster 4 times but no luck...
>>
>>
>> Any other hint?
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> Dear Vincent,
>>>
>>> You have to make sure that mapred-site is modified on every tasktracker
>>> node! Not just the master! Mapred-site can be customized if you have a
>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>> job.
>>>
>>> Regards, Dieter
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>> Hi Guys,
>>>>
>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>> but the default (2)
>>>>
>>>>
>>>>
>>>>
>>>> What I did do
>>>> ===========
>>>>
>>>> stop mapred
>>>> stop dfs
>>>>
>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>
>>>>
>>>> start dfs
>>>> start mapred
>>>>
>>>>
>>>> But the system still uses default values
>>>>
>>>>
>>>>
>>>>
>>>> Environment
>>>> =========
>>>>
>>>> Hadoop version: 1.1.2
>>>> Java version: 7
>>>>
>>>>
>>>> Mapred-site.xml
>>>> ============
>>>>
>>>> <property>
>>>> <name>mapred.system.dir</name>
>>>> <value>/hadoop/mapred/system</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.reduce.tasks</name>
>>>> <value>50</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.child.java.opts</name>
>>>> <value>-Xmx500m</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>machine1.mydomain.com:54311</value>
>>>>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Thanks guys for any pointers!!!!
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Vincent.
>>>>
>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
are final
On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com> wrote:
> you could make the values final: add <final> true </final> to make sure
> they cannot be overridden..
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
>> Hi Dieter,
>>
>> Thanks a lot for prompt reply! very much appreciated.
>>
>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>> restarted the cluster 4 times but no luck...
>>
>>
>> Any other hint?
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> Dear Vincent,
>>>
>>> You have to make sure that mapred-site is modified on every tasktracker
>>> node! Not just the master! Mapred-site can be customized if you have a
>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>> job.
>>>
>>> Regards, Dieter
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>> Hi Guys,
>>>>
>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>> but the default (2)
>>>>
>>>>
>>>>
>>>>
>>>> What I did do
>>>> ===========
>>>>
>>>> stop mapred
>>>> stop dfs
>>>>
>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>
>>>>
>>>> start dfs
>>>> start mapred
>>>>
>>>>
>>>> But the system still uses default values
>>>>
>>>>
>>>>
>>>>
>>>> Environment
>>>> =========
>>>>
>>>> Hadoop version: 1.1.2
>>>> Java version: 7
>>>>
>>>>
>>>> Mapred-site.xml
>>>> ============
>>>>
>>>> <property>
>>>> <name>mapred.system.dir</name>
>>>> <value>/hadoop/mapred/system</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.reduce.tasks</name>
>>>> <value>50</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.child.java.opts</name>
>>>> <value>-Xmx500m</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>machine1.mydomain.com:54311</value>
>>>>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Thanks guys for any pointers!!!!
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Vincent.
>>>>
>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
are final
On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <dr...@gmail.com> wrote:
> you could make the values final: add <final> true </final> to make sure
> they cannot be overridden..
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
>> Hi Dieter,
>>
>> Thanks a lot for prompt reply! very much appreciated.
>>
>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>> restarted the cluster 4 times but no luck...
>>
>>
>> Any other hint?
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>>
>>> Dear Vincent,
>>>
>>> You have to make sure that mapred-site is modified on every tasktracker
>>> node! Not just the master! Mapred-site can be customized if you have a
>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>> job.
>>>
>>> Regards, Dieter
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>>
>>> Hi Guys,
>>>>
>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>> but the default (2)
>>>>
>>>>
>>>>
>>>>
>>>> What I did do
>>>> ===========
>>>>
>>>> stop mapred
>>>> stop dfs
>>>>
>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>
>>>>
>>>> start dfs
>>>> start mapred
>>>>
>>>>
>>>> But the system still uses default values
>>>>
>>>>
>>>>
>>>>
>>>> Environment
>>>> =========
>>>>
>>>> Hadoop version: 1.1.2
>>>> Java version: 7
>>>>
>>>>
>>>> Mapred-site.xml
>>>> ============
>>>>
>>>> <property>
>>>> <name>mapred.system.dir</name>
>>>> <value>/hadoop/mapred/system</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.reduce.tasks</name>
>>>> <value>50</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>>> <value>15</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.child.java.opts</name>
>>>> <value>-Xmx500m</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>machine1.mydomain.com:54311</value>
>>>>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Thanks guys for any pointers!!!!
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Vincent.
>>>>
>>>>
>>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
you could make the values final: add <final> true </final> to make sure
they cannot be overridden..
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Dieter,
>
> Thanks a lot for prompt reply! very much appreciated.
>
> Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
> the cluster 4 times but no luck...
>
>
> Any other hint?
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> Dear Vincent,
>>
>> You have to make sure that mapred-site is modified on every tasktracker
>> node! Not just the master! Mapred-site can be customized if you have a
>> heterogeneous cluster. I usually start the dfs, then copy all the
>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>> job.
>>
>> Regards, Dieter
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>> Hi Guys,
>>>
>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>> but the default (2)
>>>
>>>
>>>
>>>
>>> What I did do
>>> ===========
>>>
>>> stop mapred
>>> stop dfs
>>>
>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>
>>>
>>> start dfs
>>> start mapred
>>>
>>>
>>> But the system still uses default values
>>>
>>>
>>>
>>>
>>> Environment
>>> =========
>>>
>>> Hadoop version: 1.1.2
>>> Java version: 7
>>>
>>>
>>> Mapred-site.xml
>>> ============
>>>
>>> <property>
>>> <name>mapred.system.dir</name>
>>> <value>/hadoop/mapred/system</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.reduce.tasks</name>
>>> <value>50</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx500m</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>machine1.mydomain.com:54311</value>
>>>
>>> </property>
>>>
>>>
>>>
>>> Thanks guys for any pointers!!!!
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Vincent.
>>>
>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
you could make the values final: add <final> true </final> to make sure
they cannot be overridden..
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Dieter,
>
> Thanks a lot for prompt reply! very much appreciated.
>
> Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
> the cluster 4 times but no luck...
>
>
> Any other hint?
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> Dear Vincent,
>>
>> You have to make sure that mapred-site is modified on every tasktracker
>> node! Not just the master! Mapred-site can be customized if you have a
>> heterogeneous cluster. I usually start the dfs, then copy all the
>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>> job.
>>
>> Regards, Dieter
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>> Hi Guys,
>>>
>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>> but the default (2)
>>>
>>>
>>>
>>>
>>> What I did do
>>> ===========
>>>
>>> stop mapred
>>> stop dfs
>>>
>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>
>>>
>>> start dfs
>>> start mapred
>>>
>>>
>>> But the system still uses default values
>>>
>>>
>>>
>>>
>>> Environment
>>> =========
>>>
>>> Hadoop version: 1.1.2
>>> Java version: 7
>>>
>>>
>>> Mapred-site.xml
>>> ============
>>>
>>> <property>
>>> <name>mapred.system.dir</name>
>>> <value>/hadoop/mapred/system</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.reduce.tasks</name>
>>> <value>50</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx500m</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>machine1.mydomain.com:54311</value>
>>>
>>> </property>
>>>
>>>
>>>
>>> Thanks guys for any pointers!!!!
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Vincent.
>>>
>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
you could make the values final: add <final> true </final> to make sure
they cannot be overridden..
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Dieter,
>
> Thanks a lot for prompt reply! very much appreciated.
>
> Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
> the cluster 4 times but no luck...
>
>
> Any other hint?
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> Dear Vincent,
>>
>> You have to make sure that mapred-site is modified on every tasktracker
>> node! Not just the master! Mapred-site can be customized if you have a
>> heterogeneous cluster. I usually start the dfs, then copy all the
>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>> job.
>>
>> Regards, Dieter
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>> Hi Guys,
>>>
>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>> but the default (2)
>>>
>>>
>>>
>>>
>>> What I did do
>>> ===========
>>>
>>> stop mapred
>>> stop dfs
>>>
>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>
>>>
>>> start dfs
>>> start mapred
>>>
>>>
>>> But the system still uses default values
>>>
>>>
>>>
>>>
>>> Environment
>>> =========
>>>
>>> Hadoop version: 1.1.2
>>> Java version: 7
>>>
>>>
>>> Mapred-site.xml
>>> ============
>>>
>>> <property>
>>> <name>mapred.system.dir</name>
>>> <value>/hadoop/mapred/system</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.reduce.tasks</name>
>>> <value>50</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx500m</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>machine1.mydomain.com:54311</value>
>>>
>>> </property>
>>>
>>>
>>>
>>> Thanks guys for any pointers!!!!
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Vincent.
>>>
>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
you could make the values final: add <final> true </final> to make sure
they cannot be overridden..
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Dieter,
>
> Thanks a lot for prompt reply! very much appreciated.
>
> Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
> the cluster 4 times but no luck...
>
>
> Any other hint?
>
>
>
>
>
>
> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com>wrote:
>
>> Dear Vincent,
>>
>> You have to make sure that mapred-site is modified on every tasktracker
>> node! Not just the master! Mapred-site can be customized if you have a
>> heterogeneous cluster. I usually start the dfs, then copy all the
>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>> job.
>>
>> Regards, Dieter
>>
>>
>> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>>
>> Hi Guys,
>>>
>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>> but the default (2)
>>>
>>>
>>>
>>>
>>> What I did do
>>> ===========
>>>
>>> stop mapred
>>> stop dfs
>>>
>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>
>>>
>>> start dfs
>>> start mapred
>>>
>>>
>>> But the system still uses default values
>>>
>>>
>>>
>>>
>>> Environment
>>> =========
>>>
>>> Hadoop version: 1.1.2
>>> Java version: 7
>>>
>>>
>>> Mapred-site.xml
>>> ============
>>>
>>> <property>
>>> <name>mapred.system.dir</name>
>>> <value>/hadoop/mapred/system</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.reduce.tasks</name>
>>> <value>50</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktacker.map.tasks.maximum</name>
>>> <value>15</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx500m</value>
>>> </property>
>>>
>>>
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>machine1.mydomain.com:54311</value>
>>>
>>> </property>
>>>
>>>
>>>
>>> Thanks guys for any pointers!!!!
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Vincent.
>>>
>>>
>>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hi Dieter,
Thanks a lot for prompt reply! very much appreciated.
Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
the cluster 4 times but no luck...
Any other hint?
On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com> wrote:
> Dear Vincent,
>
> You have to make sure that mapred-site is modified on every tasktracker
> node! Not just the master! Mapred-site can be customized if you have a
> heterogeneous cluster. I usually start the dfs, then copy all the
> mapredsites to the slave nodes, then run start-mapred.sh and then start a
> job.
>
> Regards, Dieter
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
> Hi Guys,
>>
>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>> but the default (2)
>>
>>
>>
>>
>> What I did do
>> ===========
>>
>> stop mapred
>> stop dfs
>>
>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>
>>
>> start dfs
>> start mapred
>>
>>
>> But the system still uses default values
>>
>>
>>
>>
>> Environment
>> =========
>>
>> Hadoop version: 1.1.2
>> Java version: 7
>>
>>
>> Mapred-site.xml
>> ============
>>
>> <property>
>> <name>mapred.system.dir</name>
>> <value>/hadoop/mapred/system</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>50</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.map.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx500m</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>machine1.mydomain.com:54311</value>
>>
>> </property>
>>
>>
>>
>> Thanks guys for any pointers!!!!
>>
>>
>> Best regards,
>>
>>
>> Vincent.
>>
>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hi Dieter,
Thanks a lot for prompt reply! very much appreciated.
Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
the cluster 4 times but no luck...
Any other hint?
On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com> wrote:
> Dear Vincent,
>
> You have to make sure that mapred-site is modified on every tasktracker
> node! Not just the master! Mapred-site can be customized if you have a
> heterogeneous cluster. I usually start the dfs, then copy all the
> mapredsites to the slave nodes, then run start-mapred.sh and then start a
> job.
>
> Regards, Dieter
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
> Hi Guys,
>>
>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>> but the default (2)
>>
>>
>>
>>
>> What I did do
>> ===========
>>
>> stop mapred
>> stop dfs
>>
>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>
>>
>> start dfs
>> start mapred
>>
>>
>> But the system still uses default values
>>
>>
>>
>>
>> Environment
>> =========
>>
>> Hadoop version: 1.1.2
>> Java version: 7
>>
>>
>> Mapred-site.xml
>> ============
>>
>> <property>
>> <name>mapred.system.dir</name>
>> <value>/hadoop/mapred/system</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>50</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.map.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx500m</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>machine1.mydomain.com:54311</value>
>>
>> </property>
>>
>>
>>
>> Thanks guys for any pointers!!!!
>>
>>
>> Best regards,
>>
>>
>> Vincent.
>>
>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hi Dieter,
Thanks a lot for prompt reply! very much appreciated.
Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
the cluster 4 times but no luck...
Any other hint?
On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com> wrote:
> Dear Vincent,
>
> You have to make sure that mapred-site is modified on every tasktracker
> node! Not just the master! Mapred-site can be customized if you have a
> heterogeneous cluster. I usually start the dfs, then copy all the
> mapredsites to the slave nodes, then run start-mapred.sh and then start a
> job.
>
> Regards, Dieter
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
> Hi Guys,
>>
>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>> but the default (2)
>>
>>
>>
>>
>> What I did do
>> ===========
>>
>> stop mapred
>> stop dfs
>>
>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>
>>
>> start dfs
>> start mapred
>>
>>
>> But the system still uses default values
>>
>>
>>
>>
>> Environment
>> =========
>>
>> Hadoop version: 1.1.2
>> Java version: 7
>>
>>
>> Mapred-site.xml
>> ============
>>
>> <property>
>> <name>mapred.system.dir</name>
>> <value>/hadoop/mapred/system</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>50</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.map.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx500m</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>machine1.mydomain.com:54311</value>
>>
>> </property>
>>
>>
>>
>> Thanks guys for any pointers!!!!
>>
>>
>> Best regards,
>>
>>
>> Vincent.
>>
>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by "Vincent Y. Shen" <vi...@gmail.com>.
Hi Dieter,
Thanks a lot for prompt reply! very much appreciated.
Indeed, I did change in the mapred-site.xml in ALL the nodes and restarted
the cluster 4 times but no luck...
Any other hint?
On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <dr...@gmail.com> wrote:
> Dear Vincent,
>
> You have to make sure that mapred-site is modified on every tasktracker
> node! Not just the master! Mapred-site can be customized if you have a
> heterogeneous cluster. I usually start the dfs, then copy all the
> mapredsites to the slave nodes, then run start-mapred.sh and then start a
> job.
>
> Regards, Dieter
>
>
> 2013/11/14 Vincent Y. Shen <vi...@gmail.com>
>
> Hi Guys,
>>
>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>> but the default (2)
>>
>>
>>
>>
>> What I did do
>> ===========
>>
>> stop mapred
>> stop dfs
>>
>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>
>>
>> start dfs
>> start mapred
>>
>>
>> But the system still uses default values
>>
>>
>>
>>
>> Environment
>> =========
>>
>> Hadoop version: 1.1.2
>> Java version: 7
>>
>>
>> Mapred-site.xml
>> ============
>>
>> <property>
>> <name>mapred.system.dir</name>
>> <value>/hadoop/mapred/system</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>50</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.reduce.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktacker.map.tasks.maximum</name>
>> <value>15</value>
>> </property>
>>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx500m</value>
>> </property>
>>
>>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>machine1.mydomain.com:54311</value>
>>
>> </property>
>>
>>
>>
>> Thanks guys for any pointers!!!!
>>
>>
>> Best regards,
>>
>>
>> Vincent.
>>
>>
>>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
Dear Vincent,
You have to make sure that mapred-site is modified on every tasktracker
node! Not just the master! Mapred-site can be customized if you have a
heterogeneous cluster. I usually start the dfs, then copy all the
mapredsites to the slave nodes, then run start-mapred.sh and then start a
job.
Regards, Dieter
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Guys,
>
> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
> cluster. But regardless whatever I tried, hadoop refuses to use my value
> but the default (2)
>
>
>
>
> What I did do
> ===========
>
> stop mapred
> stop dfs
>
> update the value of mapred.tasktacker.reduce.tasks.maximum
>
>
> start dfs
> start mapred
>
>
> But the system still uses default values
>
>
>
>
> Environment
> =========
>
> Hadoop version: 1.1.2
> Java version: 7
>
>
> Mapred-site.xml
> ============
>
> <property>
> <name>mapred.system.dir</name>
> <value>/hadoop/mapred/system</value>
> </property>
>
>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>50</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.reduce.tasks.maximum</name>
> <value>15</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.map.tasks.maximum</name>
> <value>15</value>
> </property>
>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx500m</value>
> </property>
>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine1.mydomain.com:54311</value>
>
> </property>
>
>
>
> Thanks guys for any pointers!!!!
>
>
> Best regards,
>
>
> Vincent.
>
>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
Dear Vincent,
You have to make sure that mapred-site is modified on every tasktracker
node! Not just the master! Mapred-site can be customized if you have a
heterogeneous cluster. I usually start the dfs, then copy all the
mapredsites to the slave nodes, then run start-mapred.sh and then start a
job.
Regards, Dieter
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Guys,
>
> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
> cluster. But regardless whatever I tried, hadoop refuses to use my value
> but the default (2)
>
>
>
>
> What I did do
> ===========
>
> stop mapred
> stop dfs
>
> update the value of mapred.tasktacker.reduce.tasks.maximum
>
>
> start dfs
> start mapred
>
>
> But the system still uses default values
>
>
>
>
> Environment
> =========
>
> Hadoop version: 1.1.2
> Java version: 7
>
>
> Mapred-site.xml
> ============
>
> <property>
> <name>mapred.system.dir</name>
> <value>/hadoop/mapred/system</value>
> </property>
>
>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>50</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.reduce.tasks.maximum</name>
> <value>15</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.map.tasks.maximum</name>
> <value>15</value>
> </property>
>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx500m</value>
> </property>
>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine1.mydomain.com:54311</value>
>
> </property>
>
>
>
> Thanks guys for any pointers!!!!
>
>
> Best regards,
>
>
> Vincent.
>
>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
Dear Vincent,
You have to make sure that mapred-site is modified on every tasktracker
node! Not just the master! Mapred-site can be customized if you have a
heterogeneous cluster. I usually start the dfs, then copy all the
mapredsites to the slave nodes, then run start-mapred.sh and then start a
job.
Regards, Dieter
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Guys,
>
> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
> cluster. But regardless whatever I tried, hadoop refuses to use my value
> but the default (2)
>
>
>
>
> What I did do
> ===========
>
> stop mapred
> stop dfs
>
> update the value of mapred.tasktacker.reduce.tasks.maximum
>
>
> start dfs
> start mapred
>
>
> But the system still uses default values
>
>
>
>
> Environment
> =========
>
> Hadoop version: 1.1.2
> Java version: 7
>
>
> Mapred-site.xml
> ============
>
> <property>
> <name>mapred.system.dir</name>
> <value>/hadoop/mapred/system</value>
> </property>
>
>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>50</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.reduce.tasks.maximum</name>
> <value>15</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.map.tasks.maximum</name>
> <value>15</value>
> </property>
>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx500m</value>
> </property>
>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine1.mydomain.com:54311</value>
>
> </property>
>
>
>
> Thanks guys for any pointers!!!!
>
>
> Best regards,
>
>
> Vincent.
>
>
>
Re: mapred.tasktacker.reduce.tasks.maximum issue
Posted by Dieter De Witte <dr...@gmail.com>.
Dear Vincent,
You have to make sure that mapred-site is modified on every tasktracker
node! Not just the master! Mapred-site can be customized if you have a
heterogeneous cluster. I usually start the dfs, then copy all the
mapredsites to the slave nodes, then run start-mapred.sh and then start a
job.
Regards, Dieter
2013/11/14 Vincent Y. Shen <vi...@gmail.com>
> Hi Guys,
>
> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
> cluster. But regardless whatever I tried, hadoop refuses to use my value
> but the default (2)
>
>
>
>
> What I did do
> ===========
>
> stop mapred
> stop dfs
>
> update the value of mapred.tasktacker.reduce.tasks.maximum
>
>
> start dfs
> start mapred
>
>
> But the system still uses default values
>
>
>
>
> Environment
> =========
>
> Hadoop version: 1.1.2
> Java version: 7
>
>
> Mapred-site.xml
> ============
>
> <property>
> <name>mapred.system.dir</name>
> <value>/hadoop/mapred/system</value>
> </property>
>
>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>50</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.reduce.tasks.maximum</name>
> <value>15</value>
> </property>
>
> <property>
> <name>mapred.tasktacker.map.tasks.maximum</name>
> <value>15</value>
> </property>
>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx500m</value>
> </property>
>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine1.mydomain.com:54311</value>
>
> </property>
>
>
>
> Thanks guys for any pointers!!!!
>
>
> Best regards,
>
>
> Vincent.
>
>
>