You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Amandeep Khurana <am...@gmail.com> on 2009/03/28 00:31:16 UTC

Reducer handing at 66%

I have a MR job running on approximately 150000 lines of data in a text
file. The reducer hangs at 66% and at that moment the cpu usage is at 100%.
After 600 seconds, it kills the job... What could be going wrong and where
should I look for the problem?


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz

Re: Reducer handing at 66%

Posted by Billy Pearson <sa...@pearsonwholesale.com>.

66% is the start of the reduce function so its likely a endless loop there 
burning the cpu cycles


"Amandeep Khurana" <am...@gmail.com> wrote in 
message news:35a22e220903271631i25ff749bx5814348e66ff4d60@mail.gmail.com...
>I have a MR job running on approximately 150000 lines of data in a text
> file. The reducer hangs at 66% and at that moment the cpu usage is at 
> 100%.
> After 600 seconds, it kills the job... What could be going wrong and where
> should I look for the problem?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>

Re: Reducer handing at 66%

Posted by schubert zhang <zs...@gmail.com>.

It seems there is a endless loop in you reduce() function.You may do not
call values.next().

On Sat, Mar 28, 2009 at 7:31 AM, Amandeep Khurana <am...@gmail.com> wrote:

> I have a MR job running on approximately 150000 lines of data in a text
> file. The reducer hangs at 66% and at that moment the cpu usage is at 100%.
> After 600 seconds, it kills the job... What could be going wrong and where
> should I look for the problem?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>