You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Eric Zhang <ez...@yahoo-inc.com> on 2007/08/20 21:31:23 UTC
how to deal with large amount of key value pair outputs in one run of map task
Hi,
I have a hadoop application where each run of the map could potentially
generate large amount of key value pairs, so it caused the run of memory
error. I am wondering if there is a way to inform hadoop to write the key
value pairs to disk periodically?
thanks,
Eric Zhang
Vespa content @Yahoo!
Work: 408-349-2466
Re: how to deal with large amount of key value pair outputs in one run of map task
Posted by Toby DiPasquale <co...@gmail.com>.
On 8/20/07, Eric Zhang <ez...@yahoo-inc.com> wrote:
> Hi,
> I have a hadoop application where each run of the map could potentially
> generate large amount of key value pairs, so it caused the run of memory
> error. I am wondering if there is a way to inform hadoop to write the key
> value pairs to disk periodically?
The output collector objects should be doing this already, right?
--
Toby DiPasquale
RE: how to deal with large amount of key value pair outputs in one run of map task
Posted by Eric Zhang <ez...@yahoo-inc.com>.
Thanks, Owen. By configing mapred.child.java.opts to larger value (took a
little while to figure out the right way to config it: -Xmx300m), the
outofmemory problem went away. It's good know that the default value of
io.sort.mb is set to 100M and my map task required about 300M heap size to
run.
Eric Zhang
Vespa content @Yahoo!
Work: 408-349-2466
-----Original Message-----
From: Owen O'Malley [mailto:oom@yahoo-inc.com]
Sent: Tuesday, August 21, 2007 10:32 AM
To: hadoop-user@lucene.apache.org
Subject: Re: how to deal with large amount of key value pair outputs in one
run of map task
On Aug 20, 2007, at 2:05 PM, Eric Zhang wrote:
> Thanks a lot for the response, Arun. Just curious how OutputCollector
> flushes key/value pair to disk: is the periodical flush based on time
> (like every couple of mins) or based on volumn (like every 100
> key/value pair output).
> The size of map output varies for each key/value input, it could be as
> small as one key/value pair output or as big as tens of millions of
> key/ value pairs. I could try to change the way my application works
> to avoid this problem, but I am wondering if the hadoop already
> supports the scalability in such case besides increasing memeory?
It uses io.sort.mb, which is the number of megabytes to keep before you sort
and spill to disk. (The config variable was named back when the sort was
being handled very differently, and thus the unobvious
name.) A major point of map/reduce is to scale to very large data sets and
make very few assumptions about what will fit in memory at once.
-- Owen
Re: how to deal with large amount of key value pair outputs in one run of map task
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Aug 20, 2007, at 2:05 PM, Eric Zhang wrote:
> Thanks a lot for the response, Arun. Just curious how OutputCollector
> flushes key/value pair to disk: is the periodical flush based on
> time (like
> every couple of mins) or based on volumn (like every 100 key/value
> pair
> output).
> The size of map output varies for each key/value input, it could be
> as small
> as one key/value pair output or as big as tens of millions of key/
> value
> pairs. I could try to change the way my application works to avoid
> this
> problem, but I am wondering if the hadoop already supports the
> scalability
> in such case besides increasing memeory?
It uses io.sort.mb, which is the number of megabytes to keep before
you sort and spill to disk. (The config variable was named back when
the sort was being handled very differently, and thus the unobvious
name.) A major point of map/reduce is to scale to very large data
sets and make very few assumptions about what will fit in memory at
once.
-- Owen
RE: how to deal with large amount of key value pair outputs in one run of map task
Posted by Eric Zhang <ez...@yahoo-inc.com>.
Thanks a lot for the response, Arun. Just curious how OutputCollector
flushes key/value pair to disk: is the periodical flush based on time (like
every couple of mins) or based on volumn (like every 100 key/value pair
output).
The size of map output varies for each key/value input, it could be as small
as one key/value pair output or as big as tens of millions of key/value
pairs. I could try to change the way my application works to avoid this
problem, but I am wondering if the hadoop already supports the scalability
in such case besides increasing memeory?
Thanks,
Eric Zhang
Vespa content @Yahoo!
Work: 408-349-2466
-----Original Message-----
From: Arun C Murthy [mailto:arunc@yahoo-inc.com]
Sent: Monday, August 20, 2007 12:58 PM
To: hadoop-user@lucene.apache.org
Subject: Re: how to deal with large amount of key value pair outputs in one
run of map task
Eric,
On Mon, Aug 20, 2007 at 12:31:23PM -0700, Eric Zhang wrote:
>Hi,
>I have a hadoop application where each run of the map could potentially
>generate large amount of key value pairs, so it caused the run of
>memory error. I am wondering if there is a way to inform hadoop to
>write the key value pairs to disk periodically?
>
The standard OutputCollector already does sort and flush key/value pairs to
disk periodically... clearly you could see memory-related issues during sort
etc.
What is the observed size of map outputs? Try increasing the child-jvm
memory limit via mapred.child.java.opts (default is 200M).
Arun
>thanks,
>
>Eric Zhang
>Vespa content @Yahoo!
>Work: 408-349-2466
>
>
Re: how to deal with large amount of key value pair outputs in one run of map task
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Eric,
On Mon, Aug 20, 2007 at 12:31:23PM -0700, Eric Zhang wrote:
>Hi,
>I have a hadoop application where each run of the map could potentially
>generate large amount of key value pairs, so it caused the run of memory
>error. I am wondering if there is a way to inform hadoop to write the key
>value pairs to disk periodically?
>
The standard OutputCollector already does sort and flush key/value pairs to disk periodically... clearly you could see memory-related issues during sort etc.
What is the observed size of map outputs? Try increasing the child-jvm memory limit via mapred.child.java.opts (default is 200M).
Arun
>thanks,
>
>Eric Zhang
>Vespa content @Yahoo!
>Work: 408-349-2466
>
>