Re: reducers hanging problem

[Andreas Kostyrka <an...@kostyrka.org>]

On Monday 30 June 2008 17:49:38 Chris Anderson wrote:
> On Mon, Jun 30, 2008 at 8:30 AM, Andreas Kostyrka <an...@kostyrka.org> 
wrote:
> >  Plus it seems to be deterministic, it always stop at 3 reduce parts
> > not finishing, although I haven't yet checked if they are always the same
> > errors or not.
>
> I've been struggling through getting my streaming tasks (map only, no
> reduce) to run across large Nutch crawls. I'd been having
> deterministic failures as well. It's worth checking your streaming job
> against input data (maybe on a local workstation) to see that it
> doesn't blow up on some of the input. It turns out my Ruby/Hpricot XML
> parsers were having a hard time swallowing large binary files
> (surprise) and as a result, some map tasks would always die in the
> same place.

I am not getting failures => failures would mean that my driver script retries 
it, ...

In my case it just stops. The external reducer program hangs while reading 
stdin, and that's it. For 5 hours or so.

I'll try something else, I won't kill these hung processes, instead I'll run a 
longer-term strace on it, and see if some data trickles into it or not.

And while my reducers output potentially long lines (in my case, it's a list 
of cookie values associated with a given site, that can get really long), 
this is on the output side of the reducer. The cookie lines come nice and 
short from the mappers.

> I got my data to test locally but running streaming jar with cat as
> it's mapper, and then copying the results to my workstation, and
> piping them into my script. I haven't tried using cat as a reducer,
> but it should yield output files suitable for running your streaming
> reducers over, in an instrumented environment.

The problem is most probably not in my reducer. Notice that hadoop, not my 
reducer, is having problems to fetch map output files. My reducer just sleeps 
like an innocent babe waiting on data.

Andreas