You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by shangan <sh...@corp.kaixin001.com> on 2010/08/17 11:47:10 UTC
reduce quite slow
I did a "select count(*) from", it's quite slow and I try to set mapred.reduce.tasks higher, but the reduce task turn out always unchanged and remain to 1(I can see it in the mapreduce administrator Web UI).
It seems the map process is quite fast but the reduce process is quite slow, the phenomenon is that the reduce can jump to 20% in a short time but then will block there for quite a long time, the following is the actuall process:
> select count(1) from log_game_farm_goods;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201008171643_0001, Tracking URL = http://vm153:50030/jobdetails.jsp?jobid=job_201008171643_0001
Kill Command = /home/shangan/bin/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=vm153:9001 -kill job_201008171643_0001
2010-08-17 16:44:27,100 Stage-1 map = 0%, reduce = 0%
2010-08-17 16:44:34,146 Stage-1 map = 40%, reduce = 0%
2010-08-17 16:44:38,169 Stage-1 map = 60%, reduce = 0%
2010-08-17 16:44:40,189 Stage-1 map = 100%, reduce = 0%
2010-08-17 16:44:43,210 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:45:43,637 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:46:44,060 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:47:44,455 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:48:44,835 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:49:45,170 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:50:45,446 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:51:45,790 Stage-1 map = 100%, reduce = 13%
2010-08-17 16:52:42,033 Stage-1 map = 100%, reduce = 20%
2010-08-17 16:52:51,070 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:53:51,410 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:54:51,669 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:55:51,933 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:56:52,321 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:57:52,605 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:58:52,913 Stage-1 map = 100%, reduce = 27%
2010-08-17 16:59:53,168 Stage-1 map = 100%, reduce = 27%
2010-08-17 17:00:53,452 Stage-1 map = 100%, reduce = 27%
2010-08-17 17:01:53,764 Stage-1 map = 100%, reduce = 27%
2010-08-17 17:02:54,008 Stage-1 map = 100%, reduce = 27%
2010-08-17 17:03:09,085 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201008171643_0001
OK
24757
Time taken: 1128.692 seconds
My cluster consists of only 4 nodes :1 namenode and 3 datanodes, it seems that the network speed is not quite good,there's too many fetch-failures in the mapreduce running. And I think this is the reason,if so can anyone tell me the request for network status, otherwise tell me the reason. Thanks a lot!
2010-08-17
shangan
Re: reduce quite slow
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Aug 17, 2010 at 5:47 AM, shangan <sh...@corp.kaixin001.com> wrote:
> I did a "select count(*) from", it's quite slow and I try to set
> mapred.reduce.tasks higher, but the reduce task turn out always unchanged
> and remain to 1(I can see it in the mapreduce administrator Web UI).
> It seems the map process is quite fast but the reduce process is quite slow,
> the phenomenon is that the reduce can jump to 20% in a short time but then
> will block there for quite a long time, the following is the actuall
> process:
>
> > select count(1) from log_game_farm_goods;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> Starting Job = job_201008171643_0001, Tracking URL = http://vm153:50030/jobdetails.jsp?jobid=job_201008171643_0001
> Kill Command = /home/shangan/bin/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=vm153:9001 -kill job_201008171643_0001
> 2010-08-17 16:44:27,100 Stage-1 map = 0%, reduce = 0%
> 2010-08-17 16:44:34,146 Stage-1 map = 40%, reduce = 0%
> 2010-08-17 16:44:38,169 Stage-1 map = 60%, reduce = 0%
> 2010-08-17 16:44:40,189 Stage-1 map = 100%, reduce = 0%
> 2010-08-17 16:44:43,210 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:45:43,637 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:46:44,060 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:47:44,455 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:48:44,835 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:49:45,170 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:50:45,446 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:51:45,790 Stage-1 map = 100%, reduce = 13%
> 2010-08-17 16:52:42,033 Stage-1 map = 100%, reduce = 20%
> 2010-08-17 16:52:51,070 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:53:51,410 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:54:51,669 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:55:51,933 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:56:52,321 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:57:52,605 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:58:52,913 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 16:59:53,168 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 17:00:53,452 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 17:01:53,764 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 17:02:54,008 Stage-1 map = 100%, reduce = 27%
> 2010-08-17 17:03:09,085 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201008171643_0001
> OK
> 24757
> Time taken: 1128.692 seconds
>
>
> My cluster consists of only 4 nodes :1 namenode and 3 datanodes, it seems
> that the network speed is not quite good,there's too many fetch-failures in
> the mapreduce running. And I think this is the reason,if so can anyone tell
> me the request for network status, otherwise tell me the reason. Thanks a
> lot!
>
> 2010-08-17
> ________________________________
> shangan
Shangan,
ORDER BY and COUNT are currently implemented by one reducer. With
this type of query you can expect the reducer to jump completion
drastically. You said the reason yourself, the reducer is fetching the
output of map tasks, since you have one reducer this can take quite a
while, particularly if you have many maps as the reducers is fetching.
There is a variable in hadoop to control max reduce fetch tasks I
believe. In any case the sum function in the final reducer is rather
trivial, so once the map output is fetched the computation is quick.
Regards,
Edward