Re: Map output in disk and memory at same time?

[Arun C Murthy <ac...@yahoo-inc.com>]

On Jan 31, 2011, at 10:51 AM, Pedro Costa wrote:

> Hi,
>
> When the reduce fetch from the mappers a map output of the size of 1GB
> and do the merge, is it possible that part of the map output is saved
> in disk and other part in memory?
>

Yes, the reduce tries to keep as much in memory as possible.

If it's under memory pressure it merges and writes to disk.

If there is excess memory it merges into memory.

Then there are 3 kinds of merges going on, depending on available  
memory:
memory-to-memory
memory-to-disk
disk-to-disk

The final merge (from memory and/or disk) feeds the 'reduce' function.

Arun