You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/09/16 16:31:33 UTC
Communication in Hadoop YARN?
Hi,
Hadoop YARN uses HDFS to store and read data from the filesystem. But
what communication technology uses to transfer data between map and
reduces, and for the node managers contact the resource manager? All
communication in point to point?
Thanks,
Re: Communication in Hadoop YARN?
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi,
** transfer data between map and reduces : *Reducer doesn't fetch the data
directly from the map there is basically shuffle copy phase where reducer
connects to the shuffle handler aux service running as part of NM through
Http to copy the Map output data.
* *node managers contact the resource manager : *This communication
basically happens through RPC using proto bufs.
Regards,
+ Naga
On Wed, Sep 16, 2015 at 8:01 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> Hadoop YARN uses HDFS to store and read data from the filesystem. But what
> communication technology uses to transfer data between map and reduces, and
> for the node managers contact the resource manager? All communication in
> point to point?
>
> Thanks,
>
Re: Communication in Hadoop YARN?
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi,
** transfer data between map and reduces : *Reducer doesn't fetch the data
directly from the map there is basically shuffle copy phase where reducer
connects to the shuffle handler aux service running as part of NM through
Http to copy the Map output data.
* *node managers contact the resource manager : *This communication
basically happens through RPC using proto bufs.
Regards,
+ Naga
On Wed, Sep 16, 2015 at 8:01 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> Hadoop YARN uses HDFS to store and read data from the filesystem. But what
> communication technology uses to transfer data between map and reduces, and
> for the node managers contact the resource manager? All communication in
> point to point?
>
> Thanks,
>
Re: Communication in Hadoop YARN?
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi,
** transfer data between map and reduces : *Reducer doesn't fetch the data
directly from the map there is basically shuffle copy phase where reducer
connects to the shuffle handler aux service running as part of NM through
Http to copy the Map output data.
* *node managers contact the resource manager : *This communication
basically happens through RPC using proto bufs.
Regards,
+ Naga
On Wed, Sep 16, 2015 at 8:01 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> Hadoop YARN uses HDFS to store and read data from the filesystem. But what
> communication technology uses to transfer data between map and reduces, and
> for the node managers contact the resource manager? All communication in
> point to point?
>
> Thanks,
>
Re: Communication in Hadoop YARN?
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi,
** transfer data between map and reduces : *Reducer doesn't fetch the data
directly from the map there is basically shuffle copy phase where reducer
connects to the shuffle handler aux service running as part of NM through
Http to copy the Map output data.
* *node managers contact the resource manager : *This communication
basically happens through RPC using proto bufs.
Regards,
+ Naga
On Wed, Sep 16, 2015 at 8:01 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> Hadoop YARN uses HDFS to store and read data from the filesystem. But what
> communication technology uses to transfer data between map and reduces, and
> for the node managers contact the resource manager? All communication in
> point to point?
>
> Thanks,
>