You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Aaron Baff <Aa...@telescope.tv> on 2011/02/25 21:38:27 UTC

Error in Reducer stderr preventing reducer from completing

I'm running into an inconsistent problem where with multiple reducers (and extremely small amounts of data) I'll hit a situation where a Reducer task attempt will hang at 33% until the JobTracker starts up a 2nd attempt. It does not seem to be tied to any 1 machine, and when I change the number of reducers to 1, it runs fine every time.



log4j:ERROR Failed to flush writer,

java.io.InterruptedIOException

        at java.io.FileOutputStream.writeBytes(Native Method)

        at java.io.FileOutputStream.write(FileOutputStream.java:260)

        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)

        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)

        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)

        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)

        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)

        at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58)

        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:316)

        at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)

        at org.apache.hadoop.mapred.TaskLogAppender.append(TaskLogAppender.java:58)

        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)

        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)

        at org.apache.log4j.Category.callAppenders(Category.java:206)

        at org.apache.log4j.Category.forcedLog(Category.java:391)

        at org.apache.log4j.Category.log(Category.java:856)

        at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:199)

        at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.freeHost(ShuffleScheduler.java:345)

        at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152)





Aaron Baff | Developer | Telescope, Inc.

email:  aaron.baff@telescope.tv<ma...@telescope.tv> | office:  424 270 2913 | www.telescope.tv<http://www.telescope.tv>


<http://twitter.com/Telescope_Inc>AMERICAN IDOL is back and better than ever with a new team of judges on Season 10! Voting begins Tuesday, March 1st so be sure to tune in and vote to make sure your favorite contestant makes it through each week (check local listings for dates & times). More info at www.americanidol.com<http://www.americanidol.com/>.



The information contained in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any views expressed in this message are those of the individual and may not necessarily reflect the views of Telescope Inc. or its associated companies.