heap memory

[Gang Luo <lg...@yahoo.com.cn>]

HI all,
I suppose there is only map function that will consume the heap memory assigned to each map task. While the default heap memory is 200 mb, I just wonder most of the memory is wasted for a simple map function (e.g. IdentityMapper). 

So, I try to make use of this memory by buffering the output records, or maintaining large data structure in memory, but it doesn't work as I expect. For example, I want to build a hash table on a 100mb table in memory during the life time of that map task. But it fails due to lack of heap memory. Don't I get 200mb heap memory? What others also eat my heap memory?

Thanks.
-Gang



      ___________________________________________________________ 
  好玩贺卡等你发，邮箱贺卡全新上线！ 
http://card.mail.cn.yahoo.com/

RE: heap memory

[Srigurunath Chakravarthi <sr...@yahoo-inc.com>]

Sorry, I sent out an incomplete message. The last sentence should be:

You can request a larger heap for Map tasks by including -Xmx
in mapred.child.java.opts

regards,
Sriguru

>-----Original Message-----
>From: Srigurunath Chakravarthi
>Sent: Monday, February 08, 2010 11:51 PM
>To: 'common-user@hadoop.apache.org'
>Subject: RE: heap memory
>
>Hi Gang,
> Not sure if I understood your question right. Responses inline:
>
>>-----Original Message-----
>>From: Gang Luo [mailto:lgpublic@yahoo.com.cn]
>>Sent: Friday, February 05, 2010 8:24 AM
>>To: common-user@hadoop.apache.org
>>Subject: heap memory
>>
>>HI all,
>>I suppose there is only map function that will consume the heap memory
>>assigned to each map task. While the default heap memory is 200 mb, I just
>>wonder most of the memory is wasted for a simple map function (e.g.
>>IdentityMapper).
>
>Map Output records get buffered before being spilt to disk. You can control
>this size via io.sort.mb.
>
>>
>>So, I try to make use of this memory by buffering the output records, or
>>maintaining large data structure in memory, but it doesn't work as I
>>expect. For example, I want to build a hash table on a 100mb table in
>>memory during the life time of that map task. But it fails due to lack of
>>heap memory. Don't I get 200mb heap memory? What others also eat my heap
>>memory?
>
>You can request a larger head for Map tasks by including -Xmx
>
>>
>>Thanks.
>>-Gang
>>
>>
>>
>>      ___________________________________________________________
>>  好玩贺卡等你发，邮箱贺卡全新上线！
>>http://card.mail.cn.yahoo.com/

RE: heap memory

[Srigurunath Chakravarthi <sr...@yahoo-inc.com>]

Hi Gang,
 Not sure if I understood your question right. Responses inline:

>-----Original Message-----
>From: Gang Luo [mailto:lgpublic@yahoo.com.cn]
>Sent: Friday, February 05, 2010 8:24 AM
>To: common-user@hadoop.apache.org
>Subject: heap memory
>
>HI all,
>I suppose there is only map function that will consume the heap memory
>assigned to each map task. While the default heap memory is 200 mb, I just
>wonder most of the memory is wasted for a simple map function (e.g.
>IdentityMapper).

Map Output records get buffered before being spilt to disk. You can control this size via io.sort.mb.

>
>So, I try to make use of this memory by buffering the output records, or
>maintaining large data structure in memory, but it doesn't work as I
>expect. For example, I want to build a hash table on a 100mb table in
>memory during the life time of that map task. But it fails due to lack of
>heap memory. Don't I get 200mb heap memory? What others also eat my heap
>memory?

You can request a larger head for Map tasks by including -Xmx

>
>Thanks.
>-Gang
>
>
>
>      ___________________________________________________________
>  好玩贺卡等你发，邮箱贺卡全新上线！
>http://card.mail.cn.yahoo.com/