number of running reducers exceed max number of mappers on cluster

[Keren Ouaknine <ke...@gmail.com>]

Pig script allocates 15 nodes to run a pig script *(how to configure
that?)*but what happens on a 10 nodes cluster only?
UI shows that 15 reducers are running, and at the same time max number of
reducers is 10 (configured one reducer per machine).



Thanks,
Keren

Kind% CompleteNum TasksPendingRunningCompleteKilledFailed/Killed
Task Attempts<http://10.239.24.12:50030/jobfailures.jsp?jobid=job_201111301137_0049>
map<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=map&pagenum=1>
100.00%
1001<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=map&pagenum=1&state=completed>
00 / 0 reduce<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=reduce&pagenum=1>
5.92%

9070<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=reduce&pagenum=1&state=pending>
15<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=reduce&pagenum=1&state=running>
5<http://10.239.24.12:50030/jobtasks.jsp?jobid=job_201111301137_0049&type=reduce&pagenum=1&state=completed>
00 / 0
Running Map TasksRunning Reduce TasksTotal Submissions NodesOccupied Map
SlotsOccupied Reduce SlotsReserved Map SlotsReserved Reduce SlotsMap Task
CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted Nodes Graylisted
NodesExcluded Nodes 004910<http://10.239.24.12:50030/machines.jsp?type=active>
000010102.000 <http://10.239.24.12:50030/machines.jsp?type=blacklisted>
0<http://10.239.24.12:50030/machines.jsp?type=graylisted>
0 <http://10.239.24.12:50030/machines.jsp?type=excluded>



-- 
 Keren Ouaknine
Cell: +972 54 2565404
Web: www.kereno.com

Re: number of running reducers exceed max number of mappers on cluster

[Harsh J <ha...@cloudera.com>]

For how to control Pig's parallelism: http://pig.apache.org/docs/r0.9.1/perf.html#parallel

For your 'running' task count bug: https://issues.apache.org/jira/browse/MAPREDUCE-486 (Which would mean that only 5 may be 'running', while the rest may have already completed and were awaiting finalization of outputs. Not to be worried about - its more of a UI bug.)

On 01-Dec-2011, at 6:36 AM, Keren Ouaknine wrote:

> Pig script allocates 15 nodes to run a pig script (how to configure that?) but what happens on a 10 nodes cluster only?
> UI shows that 15 reducers are running, and at the same time max number of reducers is 10 (configured one reducer per machine).
> 
> 
> 
> Thanks,
> Keren
> 
> Kind	% Complete	Num Tasks	Pending	Running	Complete	Killed	Failed/Killed
> Task Attempts
> map	100.00%
> 
> 1	0	0	1	0	0 / 0
> reduce	5.92%
> 
> 
> 90	70	15	5	0	0 / 0
> 
> Running Map Tasks	Running Reduce Tasks	Total Submissions	 Nodes	Occupied Map Slots	Occupied Reduce Slots	Reserved Map Slots	Reserved Reduce Slots	Map Task Capacity	Reduce Task Capacity	Avg. Tasks/Node	Blacklisted Nodes	Graylisted Nodes	Excluded Nodes
> 0	0	49	10	0	0	0	0	10	10	2.00	0	0	0
> 
> 
> 
> 
> -- 
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
> 
>