Help for Tasktracker down with getMapOutput() fails with EofException, followed by IllegalStateException

[Yogesh Sangvikar <Yo...@persistent.co.in>]

Hi Sir,

I have 4 node hadoop cluster with one of the node acting as master as well as slave node. The SecondayNameNode is running on one of the slave node.
I am using 8 map (mapred.map.tasks) and 4 reduce (mapred.reduce.tasks) tasks. I am using the cluster for Nutch 1.4 crawling.
I observed that, one of the slave node (node with secondary Name node) got down with below Exception in tasktracker log,

2012-09-21 00:37:40,819 WARN  mapred.TaskTracker - getMapOutput(attempt_201209201246_0063_m_000036_0,2) failed :
org.mortbay.jetty.EofException
       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
       at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
       at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
       at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,820 WARN  mortbay.log - Committed before 410 getMapOutput(attempt_201209201246_0063_m_000036_0,2) failed :
org.mortbay.jetty.EofException
       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
       at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
       at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
       at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,823 ERROR mortbay.log - /mapOutput
java.lang.IllegalStateException: Committed
       at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
       at org.mortbay.jetty.Response.sendError(Response.java:240)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2963)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
       at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
       at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)


I suspect, above Exceptions has caused the slave node to get down.
I found one reference on JIRA with such issue, but didn't get the root cause and way to resolve such issues.
https://issues.apache.org/jira/browse/MAPREDUCE-5

Could you help me to understand the root cause of above exceptions and whether this may cause the Tasktracker to get down?

Your comments will be helpful to understand and give hint to improve the hadoop performance.

Thanks,
Yogesh Sangvikar






DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.