Program crashed when volume of data getting large

[Kunsheng Chen <ke...@yahoo.com>]

Hi everyone,


I am running two map-reduce program, they were working good but when the data turns into around 900MB (50000+ files). things weird happen to remind me as below:


'Communication problem with server: java.net.SocketTimeoutException: timed out waiting for rpc response'

Also there is some other reminder like "fail to allocate memory".


Strange is that the program keeps running and shows mapping and reduce percentage after those errors....seems it is still progressing in a slow pace.


Does anyone have some idea ?



Thanks,

-Kun





      

Re: Program crashed when volume of data getting large

[Chandraprakash Bhagtani <cp...@gmail.com>]

your tasks are running out of memory you can increase memory by setting
property
*mapred.child.java.opts *-Xmx500m

this means your tasks (map/reduce) can use maximum of 500 MB memory. default
is 200m.
increase it as much as you physical memory allows using swap space will make
processing slow.

On Wed, Sep 23, 2009 at 6:20 PM, Kunsheng Chen <ke...@yahoo.com> wrote:

> Hi everyone,
>
>
> I am running two map-reduce program, they were working good but when the
> data turns into around 900MB (50000+ files). things weird happen to remind
> me as below:
>
>
> 'Communication problem with server: java.net.SocketTimeoutException: timed
> out waiting for rpc response'
>
> Also there is some other reminder like "fail to allocate memory".
>
>
> Strange is that the program keeps running and shows mapping and reduce
> percentage after those errors....seems it is still progressing in a slow
> pace.
>
>
> Does anyone have some idea ?
>
>
>
> Thanks,
>
> -Kun
>
>
>
>
>
>
>


-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

RE: Program crashed when volume of data getting large

[Amogh Vasekar <am...@yahoo-inc.com>]

Hi,
Please check the namenode heap usage. Your cluster may be having too many files to handle / too little free space. It is generally available in the UI. This is one of the causes I have seen for the Timeout.

Amogh
-----Original Message-----
From: Kunsheng Chen [mailto:keyek@yahoo.com] 
Sent: Wednesday, September 23, 2009 6:21 PM
To: common-user@hadoop.apache.org
Subject: Program crashed when volume of data getting large

Hi everyone,


I am running two map-reduce program, they were working good but when the data turns into around 900MB (50000+ files). things weird happen to remind me as below:


'Communication problem with server: java.net.SocketTimeoutException: timed out waiting for rpc response'

Also there is some other reminder like "fail to allocate memory".


Strange is that the program keeps running and shows mapping and reduce percentage after those errors....seems it is still progressing in a slow pace.


Does anyone have some idea ?



Thanks,

-Kun