You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Toby DiPasquale <co...@gmail.com> on 2007/08/20 21:51:40 UTC

Re: how to deal with large amount of key value pair outputs in one run of map task

On 8/20/07, Eric Zhang <ez...@yahoo-inc.com> wrote:
> Hi,
> I have a hadoop application where each run of the map could potentially
> generate large amount of key value pairs,  so it caused the run of memory
> error.  I am wondering if there is a way to inform hadoop to write the key
> value pairs to disk  periodically?

The output collector objects should be doing this already, right?

-- 
Toby DiPasquale