does reduce > copy (at 0.52 MB/s) means network or other IO problem?

[Vitaliy Semochkin <vi...@gmail.com>]

Hello,

I often see reduce > copy (at 0.52 MB/s)  phase with such speed.
Despite in my cluster all 5 nodes are in same rack.
Does it mean any network or other IO problems, or other reasons can
cause such slow speed?

Thanks in Advance,
Vitaliy S

Re: does reduce > copy (at 0.52 MB/s) means network or other IO problem?

[Harsh J <qw...@gmail.com>]

The reduce begins copying map outputs as they complete (starting at 5% of
them) and this transfer may be very meagre and thus the low rate of
transfer.

Observe once all maps finish or near completion at their last wave, if the
network status shown is still slow then there is a problem, whose common
side effect would be failing reducers or long time waits before the sort
phase kicks in even if all mappers are already done.

Otherwise this isn't an issue. You can also increase the parallel fetching
factor of each reducer :)

On Oct 5, 2010 6:49 PM, "Vitaliy Semochkin" <vi...@gmail.com> wrote:

Hello,

I often see reduce > copy (at 0.52 MB/s)  phase with such speed.
Despite in my cluster all 5 nodes are in same rack.
Does it mean any network or other IO problems, or other reasons can
cause such slow speed?

Thanks in Advance,
Vitaliy S