what does job tracker status reduce > copy mean?

[Vitaliy Semochkin <vi...@gmail.com>]

Hello,

what does job tracker's status  reduce > copy (xxx of yyy at 4.55 MB/s) mean?

Thanks in Advance,
Vitaliy S

Re: what does job tracker status reduce > copy mean?

[Vitaliy Semochkin <vi...@gmail.com>]

Thank you very much,
in my case reduce > copy (998 of 1777 at 4.55 MB/s)
freezed at number 998 and did nothing till it was aboarted as freezed task.
Does 4.55MB/s mean that data was transfered between cluster with such
small speed(i.e. a network problem)?


Once task fall with out of memory exception but same task was finished
on other node without problem.
Nodes are Identical, what could cause such behavior?

Thanks in Advance,
Vitaliy S



On Sun, Sep 26, 2010 at 8:44 AM, Neil Xu <ne...@gmail.com> wrote:
> Hi, Vitaliy
>
> a reducer has 3 phases to process data, copy(33%)=>sort(66%)=>reduce(100%).
> the information shows that the reducer is at the first phase, pulling data
> from mappers, sometimes the progress may keep 33% for a long time due to the
> limit of network bandwidth when too large data pulled to a single reducer.
>
>
> Neil
>
> 2010/9/24 Vitaliy Semochkin <vi...@gmail.com>
>>
>> Hello,
>>
>> what does job tracker's status  reduce > copy (xxx of yyy at 4.55 MB/s)
>> mean?
>>
>> Thanks in Advance,
>> Vitaliy S
>
>

Re: what does job tracker status reduce > copy mean?

[Neil Xu <ne...@gmail.com>]

Hi, Vitaliy

a reducer has 3 phases to process data, copy(33%)=>sort(66%)=>reduce(100%).
the information shows that the reducer is at the first phase, pulling data
from mappers, sometimes the progress may keep 33% for a long time due to the
limit of network bandwidth when too large data pulled to a single reducer.


Neil

2010/9/24 Vitaliy Semochkin <vi...@gmail.com>

> Hello,
>
> what does job tracker's status  reduce > copy (xxx of yyy at 4.55 MB/s)
> mean?
>
> Thanks in Advance,
> Vitaliy S
>