Re: Where is the map input transfered to the 'map worker'/container?

[Sandy Ryza <sa...@cloudera.com>]

Hi Christian,

I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of
getting the bytes from the remote node.

-Sandy


On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote <
cgrote@mail.uni-paderborn.de> wrote:

> Hey,
>
> I'm looking for the place where the actual map input is transfered to the
> 'map worker'/container (in case it's assigned to a host that doesn't have
> the data already).
>
> Something similar to the copyFromHost(..) method in Fetcher.java
> (org.apache.hadoop.mapreduce.task.reduce), where the map output is
> transfered.
>
>
> Best Regards,
> Christian Grote
>
>
>
>
>
>
>
>

Re: Where is the map input transfered to the 'map worker'/container?

[Vinayakumar B <vi...@apache.org>]

Hi Christian,

The input will be read using the InputFormat's record reader.

for example,

In case of  TexInputformat, *LineRecordReader#nextKeyValue(..)* will do the
job of reading one line from input file for every map call and storing the
offset as *key *and *text* as  *value, *and same will be passed to *map(..)*

You can go through *org.apache.hadoop.mapreduce.lib.input.LineRecordReader.java
*for more details.

Regards,
Vinay


On Mon, Jul 7, 2014 at 10:06 AM, Sandy Ryza <sa...@cloudera.com> wrote:

> Hi Christian,
>
> I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of
> getting the bytes from the remote node.
>
> -Sandy
>
>
> On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote <
> cgrote@mail.uni-paderborn.de> wrote:
>
> > Hey,
> >
> > I'm looking for the place where the actual map input is transfered to the
> > 'map worker'/container (in case it's assigned to a host that doesn't have
> > the data already).
> >
> > Something similar to the copyFromHost(..) method in Fetcher.java
> > (org.apache.hadoop.mapreduce.task.reduce), where the map output is
> > transfered.
> >
> >
> > Best Regards,
> > Christian Grote
> >
> >
> >
> >
> >
> >
> >
> >
>