You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Alexander Aristov <al...@gmail.com> on 2008/07/31 21:06:59 UTC
how to increase number of reduce tasks
Hi
I am running nutch on hadoop 0.17.1. I launch 5 nodes to perform crawling.
When I look at the job statistics I see that only 1 reduce task is stared
for all steps and hence I do a conclusion that hadoop doesn't consume all
available resources.
Only one node is extremily busy, other nodes are idle. How can I configure
hadoop to consume all resources?
I added mapred.map.tasks and mapred.reduce.tasks parameters but they have no
effect.
I also increased the max number for the mapred tasks, job tracker shows it.
During all stages map tasks reaches maximum 3, andreduce only 1.
--
Best Regards
Alexander Aristov
RE: how to increase number of reduce tasks
Posted by Chuck Lan <cl...@modeln.com>.
sorry, mapred.reduce.tasks. Maybe you need to restart the namenode?
-----Original Message-----
From: Chuck Lan [mailto:clan@modeln.com]
Sent: Thursday, July 31, 2008 12:39 PM
To: core-user@hadoop.apache.org
Subject: RE: how to increase number of reduce tasks
I just set the property map.reduce.tasks.
-Chuck
-----Original Message-----
From: Alexander Aristov [mailto:alexander.aristov@gmail.com]
Sent: Thursday, July 31, 2008 12:07 PM
To: core-user@hadoop.apache.org
Subject: how to increase number of reduce tasks
Hi
I am running nutch on hadoop 0.17.1. I launch 5 nodes to perform
crawling.
When I look at the job statistics I see that only 1 reduce task is
stared
for all steps and hence I do a conclusion that hadoop doesn't consume
all
available resources.
Only one node is extremily busy, other nodes are idle. How can I
configure
hadoop to consume all resources?
I added mapred.map.tasks and mapred.reduce.tasks parameters but they
have no
effect.
I also increased the max number for the mapred tasks, job tracker shows
it.
During all stages map tasks reaches maximum 3, andreduce only 1.
--
Best Regards
Alexander Aristov
RE: how to increase number of reduce tasks
Posted by Chuck Lan <cl...@modeln.com>.
I just set the property map.reduce.tasks.
-Chuck
-----Original Message-----
From: Alexander Aristov [mailto:alexander.aristov@gmail.com]
Sent: Thursday, July 31, 2008 12:07 PM
To: core-user@hadoop.apache.org
Subject: how to increase number of reduce tasks
Hi
I am running nutch on hadoop 0.17.1. I launch 5 nodes to perform
crawling.
When I look at the job statistics I see that only 1 reduce task is
stared
for all steps and hence I do a conclusion that hadoop doesn't consume
all
available resources.
Only one node is extremily busy, other nodes are idle. How can I
configure
hadoop to consume all resources?
I added mapred.map.tasks and mapred.reduce.tasks parameters but they
have no
effect.
I also increased the max number for the mapred tasks, job tracker shows
it.
During all stages map tasks reaches maximum 3, andreduce only 1.
--
Best Regards
Alexander Aristov