You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Naama Kraus <na...@gmail.com> on 2008/07/13 07:51:17 UTC
Map intermediate files location
Hi,
In Google's Map Reduce paper, they mention that map tasks' intermediate
files are written to local file system. Later on, they are fetched by reduce
nodes using RPC calls. Is this also true for Hadoop ?
Thanks, Naama
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: Map intermediate files location
Posted by Naama Kraus <na...@gmail.com>.
Thanks.
Do you use Jetty for the http requests ? Anything else ?
Naama
On Mon, Jul 14, 2008 at 2:04 AM, Owen O'Malley <oo...@yahoo-inc.com> wrote:
>
> On Jul 12, 2008, at 10:51 PM, Naama Kraus wrote:
>
> Hi,
>>
>> In Google's Map Reduce paper, they mention that map tasks' intermediate
>> files are written to local file system. Later on, they are fetched by
>> reduce
>> nodes using RPC calls. Is this also true for Hadoop ?
>>
>
> Yes, although we use http rather than rpc.
>
> -- Owen
>
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: Map intermediate files location
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Jul 12, 2008, at 10:51 PM, Naama Kraus wrote:
> Hi,
>
> In Google's Map Reduce paper, they mention that map tasks'
> intermediate
> files are written to local file system. Later on, they are fetched
> by reduce
> nodes using RPC calls. Is this also true for Hadoop ?
Yes, although we use http rather than rpc.
-- Owen