Hadoop Mapper Intermediate Result Storage with No Reducer

[Xun TANG <ta...@gmail.com>]

Hi,

We are running Hadoop jobs with  mappers only, no reducers.
After each map computation, it writes its result (i.e. intermediate
results) to local disk. How fast is this speed, compared with write-speed
to HDFS?
We used TestFDSIO to do the benchmark to get write-speed to HDFS, but we
wonder how much the gap is.

Another question is, when did Hadoop move this chuck to HDFS?

Any thoughts, guys? Thanks ahead.

Xun

Re: Hadoop Mapper Intermediate Result Storage with No Reducer

[Harsh J <ha...@cloudera.com>]

A map-only job does not write to local disk, but rather directly to
the configured FileSystem the job's running on (HDFS usually).

It has always been so as far as I know, as map-only jobs do not do any
form of sorting/partitioning and hence do not require transient
storage.

On Thu, Apr 26, 2012 at 2:19 AM, Xun TANG <ta...@gmail.com> wrote:
> Hi,
>
> We are running Hadoop jobs with  mappers only, no reducers.
> After each map computation, it writes its result (i.e. intermediate
> results) to local disk. How fast is this speed, compared with write-speed
> to HDFS?
> We used TestFDSIO to do the benchmark to get write-speed to HDFS, but we
> wonder how much the gap is.
>
> Another question is, when did Hadoop move this chuck to HDFS?
>
> Any thoughts, guys? Thanks ahead.
>
> Xun



-- 
Harsh J