task slots problem

[exception <ex...@taomee.com>]

Hi,

My cluster contains 5 DataNodes, each with 8 map slots and 2 reduce slots. So there are up to 40 slots in my cluster and 40 tasks can run in parallel.
But when running a particular job, I have noticed that 80 tasks running in parallel. The cluster looks fine when running other jobs.
This particular job has no reducers. Dos this relate to the problem?

Kind

% Complete

Num Tasks

Pending

Running

Complete

Killed

Failed/Killed
Task Attempts<http://master:50030/jobfailures.jsp?jobid=job_201101201639_1384>

map<http://master:50030/jobtasks.jsp?jobid=job_201101201639_1384&type=map&pagenum=1>

54.61%





1117

427<http://master:50030/jobtasks.jsp?jobid=job_201101201639_1384&type=map&pagenum=1&state=pending>

80<http://master:50030/jobtasks.jsp?jobid=job_201101201639_1384&type=map&pagenum=1&state=running>

610<http://master:50030/jobtasks.jsp?jobid=job_201101201639_1384&type=map&pagenum=1&state=completed>

0

0 / 0

reduce<http://master:50030/jobtasks.jsp?jobid=job_201101201639_1384&type=reduce&pagenum=1>

0.00%



0

0

0

0

0

0 / 0



Thanks

Re: task slots problem

[Harsh J <qw...@gmail.com>]

This is fine. 40 may be ending, while the next 40 may be starting. The tasks
will run a cleanup operation at their end (and be in such a 'status'),
during which the TaskTracker is allowed to schedule the next wave of maps it
needs. In these moments, it may appear as if 80 concurrent tasks may be
running. You can verify this by looking at the state of the 80 'running'
tasks. 40~ of them would be initializing while 40~ of them would be cleaning
up.

Reducer slots are not abused by the Mappers.

On Tue, Jan 25, 2011 at 6:33 PM, exception <ex...@taomee.com> wrote:

>  Hi,
>
>
>
> My cluster contains 5 DataNodes, each with 8 map slots and 2 reduce slots.
> So there are up to 40 slots in my cluster and 40 tasks can run in parallel.
>
> But when running a particular job, I have noticed that 80 tasks running in
> parallel. The cluster looks fine when running other jobs.
>
>
> --
Harsh J
www.harshj.com