"Timed out waiting for rpc response" after running a large number of jobs

[Kunsheng Chen <ke...@yahoo.com>]

Hi everyone,


I am running two map-reduce program, they were working good but when the data turns into around 900MB (50000+ files). things weird happen to remind me as below:


 'Communication problem with server: java.net.SocketTimeoutException: timed out waiting for rpc response'

Also there is some other reminder like "fail to allocate memory".


Strange is that the program keeps running and shows mapping and reduce percentage after those errors....seems it is still progressing in a slow pace.


Does anyone have some idea ?



Thanks,

-Kun





      

Re: "Timed out waiting for rpc response" after running a large number of jobs

[Jason Venner <ja...@gmail.com>]

It is not uncommon for the task tracker http servers to get overwhelmed with
requests for map outputs, when there where many map tasks.
Increasing the number of threads can help.

On Sat, Sep 19, 2009 at 6:32 PM, Kunsheng Chen <ke...@yahoo.com> wrote:

> Hi everyone,
>
>
> I am running two map-reduce program, they were working good but when the
> data turns into around 900MB (50000+ files). things weird happen to remind
> me as below:
>
>
>  'Communication problem with server: java.net.SocketTimeoutException: timed
> out waiting for rpc response'
>
> Also there is some other reminder like "fail to allocate memory".
>
>
> Strange is that the program keeps running and shows mapping and reduce
> percentage after those errors....seems it is still progressing in a slow
> pace.
>
>
> Does anyone have some idea ?
>
>
>
> Thanks,
>
> -Kun
>
>
>
>
>
>
>


-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals